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Preface 


The  purpose  of  this  study  was  to  investigate  problems 
in  the  data  acquisition  process.  The  study  shows  data  is  a 
cost  driver  and  the  poor  training  of  data  managers  and  lack 
of  awareness  of  program  managers  costs  the  Air  Force  over  $1 
billion  yearly  in  excess  contract  costs.  Seven  studies,  an 
auditor's  report,  and  a  survey  of  data  managers  were  used  to 
find  common  problems  and  possible  corrective  actions. 

Many  thanks  are  due  to  Ms  Brenda  Stanley  for  her 
assistance  in  securing  information  for  this  thesis  and 
Professor  Dan  Reynolds  for  his  technical  assistance.  I 
especially  thank  my  advisor.  Mr  Samuel  Epstein,  who  spent 
countless  hours  reviewing,  encouraging,  and  supporting  me  on 
this  effort.  Mr  Epstein  continues  the  battle  to  improve  the 
data  acquisition  process.  His  dedication  to  his  work  and 
his  country  were  truly  an  inspiration  that  kept  me  going  at 
those  times  when  I'd  have  rather  not. 

Those  deserving  the  most  thanks  are  my  family.  My  son, 
Tom,  and  daughter,  Lisa,  always  understood  when  Dad  was  too 
busy,  and  they  undoubtedly  sacrificed  the  most  for  this 
thesis.  My  wife,  Kathy,  has  now  survived  our  second  AFIT 
tour.  Her  encouragement,  understanding,  love,  patience  and 
sense  of  humor  were  abundant  and  continuous  throughout  this 
effort  and  without  her,  this  effort  truly  would  not  have 
been  possible.  And  finally,  I  thank  the  Lord  for  the 
strength  and  guidance  he  so  generously  bestows  upon  us. 

Daniel  C.  Brink 
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Abstract 


This  -strcrdy  investigated  the  data  management  and  data 
acquisition  area  of  the  overall  acquisition  process.  The 
investigation  looked  at  seven  studies,  an  Auditor  General's 
Report,  and  a  survey  of  data  managers  in  Air  Force  Logistics 
Command  and  Aeronautical  Systems  Division  of  Air  Force 
Systems  Command.  The  purpose  was  to  determine  what  problems 
with  data  management  and  data  acquisition  are  universal 
across  units,  divisions,  and  commands.  After  identifying 
definite  problem  areas,  the  study  probed  the  information  to 
find  what  actions  should  be  taken  to  improve  the  data 
acquisition  process.  The  problems  identifed  included  a  lack 
of  personnel  training,  poor  definition  of  data  requirements, 
and  a  proliferation  of  data  by  the  government.  The  study 
showed  that  data  is  a  significant  cost  driver  m  the 
acquisition  of  systems  and  program  managers  are  not  aware  of 
the  extent  of  those  costs.  The  data  manager  survey  was 
examined  to  determine  if  AFLC  and  ASD  data  managers  agreed 
as  to  the  problems  and  possible  solutions.  Problems  that 
permeate  the  acquisition  process  were  found,  thus  enabling 
future  policy  decisions  to  address  these  problems.  / 
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AN  ASSESSMENT  OF  THE  PROBLEMS  SURROUNDING  THE  DATA 
ACQUISITION  PROCESS,  AND  POSSIBLE  IMPROVEMENTS 


I.  What  Is  Wrong  with  the  Acquis i t ion  Process 

The  Department  of  Defense  (DOD)  must  confront  the 
challenge  of  implementing  a  new  method  to  acquire  weapon 
systems.  The  current  system  is  overwhelmed  with  schedule 
slips,  systems  that  do  not  perform  up  to  standards  after 
delivery,  cost  overruns,  and  nonsupportabi 1 i ty  after 
fielding  the  weapon  systems.  The  President  has  directed  the 
DOD  to  "...  acquire  needed  new  weapon  systems  at  less 
cost,  in  less  time,  and  with  greater  assurance  of  promised 
performance  .  .  ,  and  ensure  the  continued  strength  and 

readiness  of  the  nation's  Armed  Forces"  (9:2).  Robert 
Costello,  Assistant  Secretary  of  Defense  (Production  and 
Logistics)  declared  that  a  one-percent  improvement  in  the 
way  we  buy  weapon  systems  wt  uld  reduce  costs  by  $1.5  billion 
(7:4).  There  is  a  recognized  need  to  change  the  business  as 
usual  way  the  DOD  acquires  weapon  systems,  but  there  appear 
to  be  severe  differences  of  opinion  as  to  just  where  the 
problem  lies  and  what  corrective  actions  need  to  be  taken. 
This  chapter  attempts  to  identify  reasons  the  present 
acquisition  system  is  not  working  satisfactori ly ,  and  the 
impact  this  has  on  the  budget  and  national  security.  The 
four  areas  of  data  management,  regulations,  personnel,  and 
equipment  are  addressed. 


Data  Management 


The  current  practice  of  requesting  extensive  data  from 
contractors  represents  an  unacceptably  high  cost  to  the  DOD . 
In  1970  the  cost  of  data  that  contractors  delivered  to  the 
government  represented  a  then  unacceptable  14 %  of  the  weapon 
systems  acquisition  (22:1).  Sixteen  years  later  the  problem 
had  only  gotten  worse,  with  data  requirements  costs 
increasing  to  20%  -  50%  of  the  total  contract  price  for  a 
weapon  system  (8:10-13).  The  problem  is  not  solely  that  the 
data  requirements  placed  upon  the  contractors  are 
exorbitant,  but  sometimes  the  lack  of  data  requested  by  the 
government  also  increases  costs.  If  the  technical  data 
requested  is  not  adequate,  extensive  costs  are  incurred  in 
trying  to  recapture  the  data  necessary  for  procurement  or 
reprocurement  of  spare  parts.  Thus,  the  problem  is  not  only 
quantity  of  data  requested  of  the  contractor,  but  requesting 
the  right  data.  Three  areas  of  data  management  found  to  be 
particularly  critical  are  frequency  of  reporting, 
overspec l f icat ion .  and  transfer  of  information. 

In  a  recent  letter  to  the  President,  the  Secretary  of 
Defense  outlined  plans  to  charter  a  joint  Office  of  the 
Secretary  of  Defense  (OSD)  -  Services  task  force  to  review 
DOD 1 s  systems  acquisition.  The  task  force  is  instructed  to 
give  "...  special  scrutiny  ...  to  those  items  that 
impose  unnecessary  reports  and  reviews  on  program  offices 
and  contractors"  (9:13).  Contractors  often  spend  as  much 
time  doing  the  work  required  as  they  spend  on  the  reporting 
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of  that  work,  yet  the  government  needs  a  way  to  keep  track 
of  the  contractor's  progress  and  make  technical  assessments. 
The  government  can  take  actions  to  acquire  necessary 
information  and  relieve  the  contractor's  reporting  burden. 
Requests  for  proposal  (RFP)  should  ask  the  contractor  for 
suggestions  to  reduce  data  deliveries  and  associated  costs. 
Also,  instead  of  requiring  monthly  delivery  of  data  (which 
by  the  time  it  is  prepared  and  delivered  is  60  -  90  days 
old)  the  government  should  require  certain  technical  data  be 
presented  at  on-site  reviews  every  month,  but  not  require 
delivery  of  that  data  to  the  governmei  in  formal  reports 
(8:10-16).  This  allows  the  contractor  to  get  on  with  the 
work  instead  of  using  valuable  time  and  money  to  prepare 
formal  reports  of  what  he  has  already  told  the  government. 

Overspecification  on  the  part  of  the  government  is  also 
responsible  for  high  acquisition  costs.  Not  only  does  the 
government  overspecify  product  requirements,  it 
overspecifies  technical  data  to  be  delivered.  An  example  of 
product  overspecification  is  the  now  famous  $7600  coffee  pot 
for  the  C-5A.  Adequate  commercial  airline  coffee  makers 
were  available  for  $3000,  but  government  specifications 
required  the  coffee  pot  withstand  40  gravities  of 
acceleration  and  operate  in  the  event  of  cabin  pressure 
loss.  Consideration  was  not  given  to  the  fact  that  neither 
man  nor  aircraft  could  survive  40-Gs  and  it  is  unlikely  that 
coffee  would  be  served  if  an  emergency  such  as  loss  of  cabin 
pressure  occurred  (30:2).  Similar  overspecification  occurs 
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in  attempts  to  militarize  technical  data  supplied  to  the 
government.  For  example,  computer  and  support  equipment 
areas  can  use  commercially  available  equipment  with 
commercially  available  manuals,  yet  in  the  past,  military 
format  has  been  unnecessarily  requested,  driving  the  cost 
ever  higher  (8:10-16). 

In  an  effort  to  remedy  the  problems  of  timely  transfer 
of  the  data  requested  of  defense  contractors,  DOD  and 
industry  have  joined  in  an  effort  to  implement  a  Computer- 
Aided  Acquisition  and  Logistics  Support  System  (CALS) . 
Planners  envision  CALS  as  a  link  among  all  defense 
contractors,  all  DOD  agencies,  and  the  users  of  weapon 
systems  through  digital  exchange  networks  (6:1).  However, 
no  price  tag  has  been  attached  to  the  implementation  of  the 
highly  sophisticated  computer  system.  The  system  will 
require  extensive  security  measures  which  are  not  likely  to 
be  overcome  even  if  the  budget  would  support  acquiring  the 
system.  A  Feb  '88  GAO  report  concluded  the  security 
measures  for  a  $2.3  billion  new  computer  system  requested  by 
DOD  could  not  meet  the  security  requirements  for  use  within 
DOD  agencies  (5:31)  let  alone  networking  to  contractors.  It 
appears  any  savings  in  time  or  reduced  data  delivery  costs 
could  be  quickly  eliminated  when  the  cost  of  the  computer 
network  is  included,  though  this  is  an  early  assessment  of 
CALS. 
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Regulations 


The  problem  of  over  regulation  is  not  new,  nor  does  it 

appear  it  will  disappear  quickly.  Several  levels  of 

bureaucracy  must  be  traversed  by  the  acquisition  process, 

both  at  the  DOD  and  Congressional  levels.  The  military 

departments  and  the  Defense  Logistics  Agency  are  reviewing 

the  acquisition  process  with  an  eye  toward  moving 

responsibility  and  authority  to  the  lowest  possible  levels. 

Robert  Costello  indicates  "the  goals  are  clear;  we  must 

simplify  the  regulations'*  (7:8).  One  defense  contractor 

identifies  the  Federal  Acquisition  Regulation  (FAR)  as  the 

main  problem.  He  says  the  FAR  should  be  recognized  for  what 

it  is,  "...  thousands  and  thousands  of  pages  of  rules  and 

regulations  which  I  would  retitle  FARCE.  The  major 

ingredient  missing  from  the  regulation  is  encouragement  to 

use  good  old  common  sense"  (20:9).  Even  the  Secretary  of 

Defense  recognizes  the  seriousness  of  the  problem.  In  his 

letter  to  the  President,  Sec  Cheney  acknowledges 

.  .  .  the  (acquisition)  system  is  encumbered  by  overly 

detailed,  confusing,  and  sometimes  contradi ctory  laws, 
regulations,  directives,  instructions,  policy  memoranda 
and  other  guidances  .  .  .  Much  of  the  stifling  burden 

is  a  consequence  of  legislative  enactments,  and 
urgently  requires  attention  by  Congress.  Much  also  has 
been  administratively  imposed  and  requires  prompt 
corrective  action  by  the  DOD.  (9:12) 

Although  the  most  dominant  position  by  the  DOD  and 

defense  contractors  is  to  reduce  regulations,  this  is  not  a 

universally  endorsed  way  of  improving  the  acquisition 

process.  The  Defense  Monitor  calls  for  more  and  tighter 
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regulations  and  more  direct  involvement  by  Congress  in  the 
acquisition  process  (30:5).  Even  so,  in  an  effort  to 
streamline  the  acquisition  process,  now  cumbersome 
regulations  are  likely  to  be  reduced  and  simplified  in  the 
future . 

Personne 1 

Personnel  in  the  acquisition  system  come  under  attack 
for  two  reasons.  First,  there  are  too  many  of  them,  and 
second,  they  are  not  trained  well  enough.  The  Packard 
Commission  Report  concludes  that  implementation  of  its 
recommendations  "...  should  allow  for  a  substantial 
reduction  in  the  total  number  of  personnel  in  the  defense 
acquisition  system,  to  levels  that  compare  with  commercial 
acquisition  counterparts."  The  report  goes  on  to  warn  that 
"Eliminating  a  layer  of  management  by  moving  the  functions 
and  people  of  that  layer  to  some  other  layer  clearly  will 
not  suffice"  (9:17).  Just  how  this  reduction  in  personnel 
will  take  place  has  not  yet  been  fully  determined  and  has 
not  yet  been  officially  announced.  However,  training 
requirements  for  acquisition  personnel  are  getting 
attention.  W.  Edwards  Deming  sites  formal  training  and 
retraining  of  personnel  as  the  key  to  maintaining  a  superior 
work  force  (29:68-69).  Evidently  Sec  Cheney  agrees  as  he 
has  initiated  plans  to  declare  the  Defense  System  Management 
College  (DSMC)  as  an  intermediate  service  school,  and 
establish  a  senior  level  service  school  comparable  to  the 
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National  War  College  with  a  curriculum  to  train  senior 
acquisition  managers  (9:15).  This  will  necessitate  a  change 
in  the  career  paths  of  military  personnel.  It  has  generally 
been  thought  advantageous  to  have  senior  military  officers 
move  from  the  operational  arena  to  the  acquisition  field. 
However,  the  training  requirements  of  a  professional 
acquisition  force  will  prevent  a  military  officer  from 
attaining  the  level  of  expertise  required  in  both  the 
operational  and  acquisition  fields  (9:15).  Thus,  it  appears 
acquisition  reform  will  not  only  displace  many  jobs,  it  will 
alter  the  time  honored  tradition  of  placing  senior 
operational  military  officers  into  acquisition  jobs  by 
creating  an  acquisition  career  field  with  qualifications 
which  will  almost  certainly  exclude  operational  experience. 

Technical  Risk  and  the  Budget 

Uncertainties  in  the  budget,  and  program  cost  overruns 
have  forced  the  acquisition  process  into  a  react  rather  than 
act  mode.  Usually  this  reaction  comes  in  the  form  of 
adjusting  production  schedules,  eliminating  prototypes,  and 
sacrificing  reliability  and  maintainability  for  performance 
or  perceived  performance.  When  the  budget  axe  falls, 
programs  are  seldom,  if  ever,  canceled.  Budget  shortfalls 
are  dealt  with  by  delaying  production  schedules  or  reducing 
the  overall  number  of  units  purchased.  In  an  effort  to 
acquire  as  many  of  a  unit  as  possible,  the  budget  for  spare 
parts  is  often  reduced  or  eliminated  in  favor  of  more 
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complete  units.  Thus  a  weapon  system  is  often  fielded 
without  enough  spare  parts  to  maintain  initial  readiness 
(30:4).  Adding  to  this  problem  is  the  perception  that 
eliminating  prototypes  and  jumping  to  full  production  will 
save  money.  This  usually  results  in  systems  being  fielded 
with  flaws  that  require  expensive  retrofit  and  reduced 
performance  as  well  as  reduced  reliability  and 
maintainability.  The  quest  for  performance  usually  survives 
all  else.  Program  managers  know  promised  performance  and 
"great  technological  leaps"  are  what  sell  programs. 

Managers  believe  they  must  preserve  the  most  "glamorous" 
portion  of  their  program,  i.e.  performance,  and  will  cut 
everything  else  in  order  to  preserve  performance.  This 
attitude  usually  works  to  the  detriment  of  reliability, 
maintainability,  spare  parts,  and  operational  readiness 
(30:4).  While  this  problem  is  usually  blamed  on  the  budget, 
it  is  only  a  part  of  the  problem  here.  The  problem  appears 
to  be  primarily  a  misguided  quest  for  program  preservation 
no  matter  the  quality  of  the  product. 

Summary 

The  acquisition  process  currently  in  place  needs  a 
drastic  overhaul.  The  President,  high  DOD  officials,  and 
the  defense  industry  have  acknowledged  the  problem.  The 
process  will  not  be  changed  easily.  Jobs  will  be  lost,  more 
rigorous  training  requirements  will  be  mandated,  and  new 
career  paths  will  upset  tradition.  If  such  drastic  change 
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is  to  take  place,  total  commitment  and  a  change  of  outlook 
and  attitude  must  be  made  within  and  throughout  DOD. 
Piecemeal  changes  will  not  work.  A  new  philosophy  is 
required.  A  new  culture  must  be  established  within  DOD. 
Several  areas  have  been  targeted  to  receive  immediate  and 
drastic  attention.  Some,  perhaps  many,  of  these  reforms  may 
charge  down  the  old  familiar  path  of  fantastic  promises  with 
little  "real"  results.  The  reforms  must  be  thoroughly 
dissected  in  advance  of  implementation  to  ensure  any  "across 
the  board"  changes  will  lead  to  a  more  efficient  acquisition 
system  which  fields  quality  systems,  thus  contributing  to 
vastly  improving  the  combat  readiness  of  our  national 
defense . 
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1 1 .  The  Specific  Problem  and  How  to  Invest jqate 

While  the  acquisition  process  is  plagued  with  the  four 
problems  areas  noted  in  Chapter  One,  a  program  manager  has 
the  power  to  improve  the  process  in  all  of  these  areas. 

Each  of  these  problem  areas  can  be  addressed  in  varying 
degrees  if  the  program  manager  is  aware  of  the  type  of  data 
and  the  data  reporting  requirements  of  his  contract.  The 
program  manager  needs  to  understand  and  determine  the  cost 
impact  to  his  program  resulting  from  data  requirements.  In 
order  to  structure  the  new  acquisition  process  to  be  more 
efficient,  it  is  essential  that  those  in  the  acquisition 
arena  understand  the  impact  of  data  requirements  on  program 
cost  and  performance. 

The  Specific  Problem 

Inherent  in  the  way  the  DOD  currently  acquires  weapon 
systems  is  the  voluminous  amount  of  paperwork  generated 
during  the  acquisition  process.  This  paperwork  is  a  result 
of  the  data  requirements  associated  with  the  acquisition 
process.  Data  requirements  are  various  reports  due  to  the 
DOD  (as  determined  by  the  DOD)  during  the  course  of  the 
acquisition  process.  In  1970,  data  requirements  represented 
a  then  unacceptable  14 %  of  the  weapon  systems  acquisition 
cost  (8:1).  By  1986,  data  requirements'  costs  increased  to 
20%  -  50%  of  the  total  contract  price  for  a  weapon  system 
(6:10-13).  Why  has  the  price  paid  for  data  grown  to  such  a 
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large  part  of  the  acquisition  budget?  How  will  the  new 
acquisition  system  address  the  cost,  quality,  and  quantity 
of  data  and  data  reporting  requirements? 

Tentative  Hypotheses  and  Investigative  Questions 

The  reason  data  has  become  such  a  large  portion  of  the 
cost  of  the  acquisition  process  could  be  as  simple  as 
program  managers  not  realizing  how  much  the  data 
requirements  they  place  on  contract  cost  their  program.  Or. 
the  problem  could  be  rooted  in  the  basic  way  the  DOD  does 
business,  continually  stressing  meeting  minimum  requirements 
and  lowest  cost,  but  reporting  everything  thereby 
sacrificing  quality  for  quantity.  My  hypotheses  are  that: 
(1)  program  managers  do  not  know  how  much  data  requirements 
actually  cost  their  program,  (2)  persons  program  managers 
appoint  to  manage  data  are  poorly  trained  and  not  in  a 
position  to  exercise  real  authority  over  ordering  data,  and 
(3)  past  efforts  to  bring  data  costs  under  control  have 
failed  due  to  a  lack  of  a  consensus  and  a  lack  of 
identification  as  to  the  real  problems  surrounding  data 
management.  Questions  needing  answers  include:  (1)  Are 
program  managers  aware  of  data  requirement  costs?  and  (2) 
What  can  be  done  to  ensure  data  reporting  requirements  are 
adequately  considered  and  regulated  by  persons  in  positions 
of  authority  in  the  acquisition  process? 
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The  Particular  Investigative  Methodology 


The  exploration  phase  of  this  research  followed  two 
courses.  .  First,  a  literature  review  was  done  (Chapter  One) 
to  determine  if  the  acquisition  process  had  been  addressed 
recently,  what  was  said  about  it,  who  said  it,  and  who  is 
listening.  The  search  for  material  included  journals,  DOD 
reports.  Air  Force  Institute  of  Technology  (AFIT)  continuing 
education  materials,  and  Air  Force  and  DOD  publications. 

The  purpose  of  this  initial  search  was  to  find  a  problem 
area(s)  within  the  acquisition  process.  The  search  revealed 
that  the  acquisition  process  was  hindered  in  part  by  the 
amount  of  paperwork,  the  inadequate  training  of  people,  the 
number  of  and  conflicting  regulations,  and  poor  definition 
of  requirements  in  Requests  for  Proposals  (RFPs)  and 
Statements  of  Work  (SOWs) .  A  common  thread  in  these  four 
areas  is  data.  The  acquisition  of  and  handling  of  data 
impacts  each  of  these  areas.  Reaching  this  point,  the 
investigation  turned  to  yet  another  source  of  information. 

Identifying  the  specific  areas  of  the  data  acquisition 
process  as  a  major  problem  area  in  the  overall  acquisition 
process  was  not  new.  Permissions  were  obtained  to  re¬ 
examine  within  this  thesis  three  recent  studies:  an  Auditor 
General's  audit  (1988),  a  DOD  wide  survey  (1989),  and  an  Air 
Force  Systems  Command  (AFSC)  survey  (1990)  addressing  data. 
Some  of  the  studies  were  conducted  at  the  Air  Force  2-Letter 
level,  while  others  were  sanctioned  by  major  commands  or  the 
DOD.  Selected  studies  are  reviewed  in  Chapter  Three,  along 
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with  a  summary  of  the  findings  from  each.  The  DOD  survey 
titled  "Career  Series  Project  for  Data  and  Configuration 
Management  Personnel  Within  the  DOD"  concentrated  on  data 
managers,  their  many  different  career  series  and  a  counting 
of  total  people  involved.  The  results  of  this  DOD  survey 
are  in  Chapter  Three.  The  Auditor  General's  report 
addressed  the  limited  pricing  of  data,  the  uneconomical 
acquisition  of  data  and  the  high  cost  of  ordering 
unnecessary  data.  Another  survey,  commissioned  by  Major 
General  Teal,  Deputy  Chief  of  Staff,  Requirements, 
Headquarters  Air  Force  Systems  Command,  addressed  problems 
and  possible  improvements  in  the  acquiring  and  managing  of 
data.  The  Teal  survey  was  part  of  a  larger  effort  to 
address  improving  the  overall  acquisition  process.  A 
description  of  the  analytical  procedures  used  on  the  DOD 
survey,  the  auditor's  report,  and  the  Teal  survey  follows  in 
this  chapter.  The  analytical  results  can  be  found  in 
Chapter  Four. 

The  Career  Series  Pro  ject  for  Data  and  Configuration 

Management  Personne 1  Within  the  DoD  Survey .  This  DOD 

wide  survey  was  sponsored  by  Mr.  Peter  Yurcisin  the  Acting 
Deputy  Assistant  Secretary  of  Defense  for  Total  Quality 
Management  and  for  Standardization  and  Data  Management. 

This  survey  was  used  to  address  the  job  status  of  data 
managers,  configuration  managers,  and  engineering  data 
managers  in  DOD  to  determine  if  the  concerns  over  no 
distinct  career  ladders  for  these  people  were  valid. 
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Because  Defense  Secretary  Cheney  and  the  Packard  Commission 
desire  the  formation  of  an  Acquisition  Work  Force  (9:15-16), 
which  the  present  acquisition  process  lacks,  there  is 
concern  over  how  to  define,  identify,  and  include  these 
people  in  the  acquisition  work  force.  The  survey,  hereafter 
referred  to  as  the  DM/CM/EDM  survey,  investigated  the  tri¬ 
service  career  levels  of  data  managers.  Jobs,  titles,  and 
head  counts  by  service,  as  well  as  percent  of  job  time  spent 
on  data  management,  configuration  management,  and 
engineering  data  management  were  used  as  a  measure  of  the 
number  of  people  utilized  within  each  military  service  to 
weigh  the  importance  placed  on  the  management  of  data  within 
the  DOD.  Simple  frequencies  and  averages  were  used  to 
determine  if  trends  existed  between  the  career  series  used 
and  the  number  of  people  responsible  for  managing  data.  The 
Air  Force  used  2400  people;  the  Army  600  people;  and  the 
Navy  500  people  and  nearly  40  different  career  series.  The 
reason  so  many  career  series  had  to  be  surveyed  was  that 
there  is  no  distinct  series  for  data,  conf igurat ion ,  or 
engineering  data  management  (14). 

The  Auditor  Genera  1 ' s  Report .  This  report 
investigated  the  cost  of  unessential  and  uneconomi ca 1 ly 
acquired  data.  The  report  included  32  contracts  from  the 
Aeronautical  Systems  Division  (ASD),  Armament  Division  (now 
Munition  Systems  Division,  MSD) ,  Electronic  Systems  Division 
(ESD) ,  Ballistic  Systems  Division  (BSD)  and  Space  Systems 
Division  (SSD).  Information  available  included  total 
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contract  price,  total  price  (or  estimates)  of  data  items, 
and  an  in-depth  look  at  unessential  or  uneconomical ly 
acquired  data  items  obtained  from  a  sample  of  data  items 
from  each  contract  (3) .  The  problems  encountered  in 
compiling  and  interpreting  this  report  demonstrate  why  at  is 
so  difficult  to  address  the  data  acquisition  process.  These 
are  discussed  in  Chapter  Four.  This  thesis  analyzes  the 
report  by  using  frequencies  of  instances  of  unessential  and 
uneconomical  data  to  determine  how  common  these 
circumstances  are.  Averages  of  costs  of  data  versus 
contract  value  and  costs  of  unessential  and  uneconomical 
data  versus  total  data  costs  are  also  used  to  analyze  the 
potential  magnitude  of  the  waste  '  ita  acquisition. 

The  Teal  Survey .  TVJs  Air  Force  survey  conducted 
by  Headquarters  AFSC  was  requested  by  Major  General  David  J. 
Teal,  Deputy  Chief  of  Staff,  Requirements.  The  survey, 
hereafter  referred  to  as  the  Teal  Survey,  was  part  of  an 
effort  to  determine  a  way  of  improving  the  request  for 
proposal  (RFP)  process.  Within  the  broader  effort  to 
improve  the  RFP  process,  the  tailoring  of  data,  data  and 
data  calls  were  identified  as  critical  areas  to  be 
addressed.  The  Teal  survey  addressed  those  issues  related 
to  the  critical  area  of  data  by  questioning  people  in  two 
subject  areas:  1.  problems  with  the  current  method  used  in 
the  tailoring  of  data,  data  call,  and  the  data  acquisition 
process,  and  2.  possible  improvements  to  the  data 
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acquisition  process  (28)  .  The  raw  data  from  the  Teal  survey- 
can  be  found  in  Appendix  A. 

Teal ,  Part  One .  The  first  part  of  the  survey 
gathered  information  on  data  acquisition  problems.  Those 
surveyed  were  asked  to  evaluate  17  problem  statements.  Each 
statement  was  evaluated  with  the  following  Likert  scale: 

STRONGLY  AGREE  NEUTRAL  DISAGREE  STRONGLY 

AGREE  DISAGREE 

A  response  of  AGREE  to  a  problem  statement  meant  the 
respondee  thought  the  statement  identified  a  problem  in  the 
data  acquisition  process.  Likewise,  a  response  of  DISAGREE 
meant  the  respondee  felt  that  the  statement  did  not  identify 
a  problem  in  the  data  acquisition  process.  The  responses  of 
STRONGLY  AGREE  and  STRONGLY  DISAGREE  were  used  to  allow 
respondees  to  indicate  an  increased  emphasis  to  their 
response.  A  neutral  response  indicated  the  respondee 
acknowledged  the  statement  condition  or  situation  existed, 
but  he/she  could  not  determine  whether  or  net  the  condition 
represented  a  problem.  An  abstention  from  giving  a  response 
to  any  problem  statement  was  interpreted  to  mean  the 
respondee  was  not  familiar  with  the  condition  or  situation 
identified  in  the  statement.  The  17  problem  statements  are 
listed  in  Appendix  B  (28). 

Tea  1 ,  Part  Two .  In  the  second  part  of  the  Teal 
survey,  respondees  were  presented  with  30  statements,  each 
suggesting  an  action  aimed  at  improving  the  data  acquisition 
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process.  The  improvement  statements  can  be  found  in 
Appendix  C.  Following  each  improvement  was  a  five-point  Likert 
scale  where  the  respondee  could  rate  the  importance  of  the 
suggested  improvement.  The  following  Likert  scale  was  used. 

VERY  HIGH  HIGH  MODERATE  LOW  NONE 

A  response  of  HIGH  was  interpreted  to  mean  the  respondee 
thought  the  suggested  improvement  would  improve  the  process 
and  should  be  implemented  if  possible.  Likewise,  a  response 
of  LOW  was  interpreted  to  mean  the  respondee  doubted  the 
ability  of  the  suggested  improvement  to  improve  the  process. 

The  response  of  VERY  HIGH  indicated  the  respondee  thought 
the  suggested  improvement  should  be  implemented  without 
delay,  while  a  response  of  NONE  indicated  the  respondee 
could  find  no  benefit  from  the  suggested  improvement.  The 
response  of  MODERATE  indicates  the  respondee  acknowledges 
only  marginal  potential  benefits.  An  abstention  from  giving 
a  response  to  the  improvement  statements  was  interpreted  to 
mean  the  respondee  was  not  familiar  enough  with  the 
suggested  action  to  determine  whether  or  not  the  suggested 
action  would  result  in  an  improved  process  (28). 

Teal  Analysis  Overview .  The  Teal  survey  consisted  of 
two  parts,  data  acquisition  process  problem  identification 
and  potential  improvements.  The  survey  originators  treated 
the  survey  responses  as  a  whole  and  reported  frequency 
distributions,  thus  ranking  the  problems  and  improvements. 

A  summary  of  their  findings  can  be  found  in  Appendix  D.  The 
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survey  results  are  re-examined  in  this  thesis  in  a  new 
perspective.  While  the  survey  contained  responses  from 
AFLC,  ASD,  BSD,  ESD,  SSD,  HQ  AFSC ,  and  contractors AFLC  and 
ASD  provided  the  most  responses,  38  each.  Since  these 
combined  76  responses  accounted  for  70%  of  the  total 
responses,  and  since  the  number  of  responses  from  any 
remaining  organizations  were  small,  only  those  responses 
from  AFLC  and  ASD  are  considered.  The  areas  of  strong 
agreement  between  AFLC  and  ASD  are  emphasized,  since 
agreement  between  the  two  on  problems  or  improvements  should 
impact  the  complexion  and  direction  of  the  new  acquisition 
work  force.  The  analysis  of  the  Teal  survey  also  determined 
differences  in  AFLC  and  ASD  responses  to  the  survey 
questions.  Finding  differences  in  the  perceptions  could 
reveal  the  need  for  different  regulations,  training,  and 
upper  level  management  attention  between  AFLC  and  ASD. 

Since  this  thesis  attempts  to  uncover  not  only  what,  but  why 
problems  in  data  acquisition  exist,  it  is  appropriate  to 
determine  if  AFLC  and  ASD  agree  or  disagree  on  what  they 
perceive  to  be  the  problems  and  possible  solutions.  The 
analysis  attempts  to  find  those  problem  areas  that  both 
organizations  find  severe.  Also,  the  analysis  strives  to 
find  those  areas  of  improvement  where  both  organizations 
believe  would  be  the  most  beneficial.  After  identifying 
those  most  severe  problems  and  the  "best"  improvements,  the 
analysis  attempts  to  determine  if  the  improvements  thought 
to  be  the  most  beneficial  correlate  with  the  problems.  It 
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is  important  that  this  correlation  exist,  or  the  most  severe 
problems  may  not  be  addressed  by  the  most  beneficial 
improvements,  and  implementation  of  the  suggested 
improvements  could  very  well  result  in  treatment  of  symptoms 
instead  of  problems. 

Analytical  Concepts  and  Methods  Employed 

The  Auditor's  report  and  the  Teal  survey  used  for  this 
research  required  a  variety  of  analytical  methods.  The 
various  procedures  and  techniques  are  described  below. 

Median .  The  median  is  the  middle  value  in  a  list  of 
sample  values  when  those  values  are  arranged  in  increasing 
order.  If  the  number  of  sample  values  is  odd,  then  the 
median  is  the  single  middle  value.  If  the  sample  value  is 
even,  the  median  is  the  average  of  the  two  middle  values. 

The  median  is  not  sensitive  to  sample  values  that  may  be 
large  or  small  when  compared  to  other  sample  values  (11:15- 
16).  For  purposes  of  this  thesis,  the  median  will  be  used 
to  rank  survey  questions.  In  Chapter  Four,  the  Teal  survey 
analysis  ranks  the  responses  by  how  strong  tne  responses 
were.  For  example,  in  the  Teal  survey  it  was  desirable  to 
determine  which  improvement  the  respondees  felt  would 
benefit  the  most.  However,  when  the  medians  were  examined, 
several  survey  questions  had  the  same  median.  So,  a  method 
of  sorting  the  medians  was  used  in  this  thesis.  If  two 
questions  had  a  median  -  4,  then  the  number  of  responses 
"below  yet  equal  to"  the  median  became  the  sorting 
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discriminator.  This  is  most  easily  demonstrated  with  an 
example.  Suppose  two  sets  of  values  are  as  listed  below: 

Set  A  Set  B 

5  4 

4  4 

4*  4* 

3  4 

2  4  *  median  response 

Both  sets  have  the  same  median  -  4.  but  the  method  of 
sorting  used  for  this  thesis  ranks  Set  B  higher  than  Set  A 
because  the  number  of  responses  "below  yet  equal  to"  the 
median  in  Set  B  is  two,  while  it  is  zero  for  Set  A.  i.e. 
There  are  two  fours  below  the  median  in  Set  B,  and  there  are 
no  fours  below  the  median  in  Set  A. 

Percentages .  Percentages  serve  two  purposes  in 
presenting  and  analyzing  data.  First,  they  bring  all 
numbers  within  a  range  of  0  to  100.  Second,  percentages 
translate  data  into  a  standard  form,  with  a  base  of  100, 
enabling  relative  comparisons  (13:327).  In  their  book 
Marketing  Research .  Harper  W.  Boyd  and  Ralph  Westfall 
caution  the  researcher  on  the  misuse  of  percentages.  They 
suggest  the  following  rules  which  are  observed  in  this 
paper . 

1.  Percentages  cannot  be  averaged  unless  weighted 
by  the  size  of  the  group  from  which  it  is  derived. 

This  demands  the  use  of  weighted  averages. 

2.  If  using  too  small  of  a  base,  percentages  can 
hide  the  base  from  which  they  have  been  computed.  e.g. 
When  contrasting  60%  to  30%,  a  sizable  difference  is 
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indicated.  However,  if  there  are  only  three  case  in 
one  category  and  ten  in  another,  the  differences  may 
not  be  significant  (13:327). 

Likert  Scales .  The  use  of  a  Likert  scale  is  common  in 
surveys  like  the  Teal  survey.  The  Likert  scale  can  be  used 
with  a  large  number  of  statements  that  meet  two  criteria. 
First,  each  statement  used  in  the  survey  must  be  relevant  to 
the  attitude  being  studied.  Second,  each  statement  is 
believed  to  reflect  a  favorable  position  on  that  attitude 
(13:256) . 

The  Wi 1 coxon  Signed-Rank  Test.  Part  of  the  analysis  of 
the  Teal  survey  involves  determining  if  the  responses  to  the 
problem  statements  are  significantly  different  from  the 
neutral  response.  This  is  important,  since  an  AFLC  or  ASD 
median  response  of  neutral  reveals  little  information. 
However,  knowing  that  the  AFLC  and/or  ASD  responses  vary 
significantly  from  neutral  suggests  a  decisive  group 
response.  The  Wilcoxon  signed-rank  test  is  a  nonparametr ic 
test  that  can  be  used  to  test  for  differences  in  the  median 
responses  of  two  groups.  AFLC  and  ASD  responses  will  each 
be  tested  for  a  Median  =  3.  The  Wilcoxon  signed-rank  test 
requires  the  data  be  a  random  sample  from  a  continuous  and 
symmetric  probability  distribution  (11:602).  For  purposes 
of  this  research,  the  data  will  be  assumed  to  be  continuous 
and  symmetric.  The  Wilcoxon  signed-rank  test  efficiently 
tests  the  Null  Hypothesis,  Ho,  that  the  sample  median 
equals  3.  The  software  package  STATISTIX  (25:6.7)  easily 
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computes  the  Wilcoxon  signed-rank  test  and  is  used  for  that 
purpose  in  this  thesis.  Also,  for  purposes  of  this  thesis, 
any  responses  not  significantly  different  from  neutral  will 
be  treated  as  neutral  responses. 

The  Ko lmogorov-Smirnov  Test .  The  Kolmogorov-Smirnov 
(K-S)  test  is  used  to  determine  if  the  distributions  (spread 
and  shape)  of  two  samples  are  the  same.  This  test  is  done 
to  satisfy  one  of  the  conditions  of  the  Mann-Whitney  or 
Wilcoxon  Rank-Sum  tests  discussed  later.  STATISTIX  easily 
computes  a  P-value  for  the  K-S  test,  and  will  be  used  for 
all  K-S  calculations  (25:4.6).  For  this  thesis,  a  P-vaiue 
less  than  0.20  will  indicate  a  significant  difference  in 
sample  distributions.  For  the  analysis  in  this  thesis,  a 
level  of  significance  of  0.20  is  sought.  Thus,  an  alpha  of 
0.20  is  used  and  any  P-value  <  0.20  will  result  in  rejecting 
the  null  hypothesis  (11:308-313).  In  other  words,  if  the 
null  hypothesis  is  rejected,  there  is  80%  confidence  that 
the  null  hypothesis  is  wrong  and  was  rightly  rejected.  For 
example,  the  distributions  of  the  responses  to  problem 
statement  one  by  AFLC  are  compared  to  the  distribution  of 
the  responses  to  problem  statement  one  by  ASD .  If  the  null 
hypothesis  is  that  two  distributions  are  the  same  and  the 
K-S  test  yields  a  P-value  exceeding  0.20,  the  distributions 
are  considered  to  be  the  same.  That  is,  there  is  not  a 
significant  difference  between  the  AFLC  and  ASD  response 
distributions.  Likewise  if  the  P-value  is  less  than  0.20, 
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the  null  hypothesis  is  rejected  and  the  distributions  are 
declared  different.  The  K-S  test  results  are  in  Appendix  E. 

The  Wi 1 coxon  Rank-Sum  Test .  The  Wilcoxon  rank-sum  test 
is  a  nonparametr ic  test  used  in  this  thesis  to  determine  if 
the  medians  of  the  two  samples  are  significantly  different. 
The  Wilcoxon  rank-sum  test  is  sometimes  referred  to  as  the 
Mann-Whitney  test.  While  these  are  different  tests,  they 
lead  to  identical  results  (25:6.12).  The  rank-sum  test 
assumes  two  independent  random  samples  from  continuous 
distributions.  The  sample  distributions  must  have  the  same 
shape  and  spread,  with  only  the  medians  or  the  means  being 
allowed  to  be  different  (11:609).  Because  of  ease  of  use, 
SPSS-X  is  used  to  perform  the  Wilcoxon  rank-sum  test. 

SPSS-X  also  computes  a  2-tailed  P-value  (26:809-834).  For 
purposes  of  this  thesis,  a  P-value  less  than  0.20  indicates 
a  significant  difference  in  sample  responses.  Thus,  any 
survey  question  having  a  rank-sum  test  P-value  less  than 
0.20  indicates  AFLC  and  ASD  significantly  differ  in  their 
responses . 

Reliability.  The  data  were  tested  for  reliability 
using  the  reliability  test  available  on  SPSS-X  (26:857-72). 

A  reliability  test  (sometimes  referred  to  as  a 
reproducibility  test)  is  an  application  of  correlational 
analysis  used  to  determine  if  the  responses  to  survey 
questions  can  be  replicated.  If  responses  cannot  be 
replicated,  the  responses  are  unstable  and  of  little  use  to 
the  researcher.  This  test  is  ideally  used  to  determine  the 
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reliability  of  successive  responses  in  a  test-retest 
situation  (19:217-18).  That  is,  measure  the  responses  to  a 
survey,  then  compare  these  first  responses  to  a  later  set  of 
responses  obtained  from  giving  the  survey  again.  Since  the 
Teal  survey  was  only  given  once,  a  random  split-half 
technique  know  as  Cronbach's  Coefficient  Alpha  was  used  to 
determine  reliability.  This  technique  randomly  splits  the 
sample  responses  into  two  groups  and  measures  reliability 
(13:98-100).  Two  reliability  tests  (Cronbach's  alpha)  were 
performed,  one  on  the  responses  to  the  problem  statements 
and  another  on  the  responses  to  the  suggested  improvements. 
The  results  are  reported  in  Chapter  Four. 

Val idity .  Validity  is  the  extent  to  which  the 
measurements  made  actually  measure  what  they  are  intended  to 
measure.  For  a  set  of  measurements  to  show  validity,  they 
must  first  be  reliable,  since  validity  cannot  exist  without 
reliability  (19:219).  Validity  takes  two  major  forms, 
internal  and  external.  External  validity,  the  ability  to  be 
generalized  across  persons,  settings,  and  time  (13:94-5),  is 
examined  in  this  thesis.  External  validity  will  be  assessed 
by  comparing  the  findings  of  the  Teal  survey  to  the  findings 
of  the  studies  reviewed  in  Chapter  Three.  If  the  survey  and 
the  studies  reveal  similar  findings,  the  Teal  survey  will 
demonstrate  its  validity.  While  this  may  be  a  less  than 
rigorous  correlational  analysis  to  determine  validity,  it 
will  nevertheless  demonstrate  agreement  (or  lack  thereof)  as 
to  the  problems  affecting  the  data  acquisition  arena. 
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Correlational  Analysis .  Correlational  analysis  allows 
the  statistical  analysis  to  go  beyond  the  central  tendency 
and  variation  that  one  is  limited  to  in  an  analysis 
involving  only  one  variable.  When  using  two  variables,  an 
assessment  of  the  association,  if  any  exists,  can  be  made 
(19-195).  For  example,  it  might  be  beneficial  to  know  if 
persons  managing  data  in  AFLC  believe  the  same  data  related 
problems  exist  as  do  persons  in  ASD . 

Factor  Analysis .  The  Teal  survey  was  designed  to 
collect  information  on  a  wide  variety  of  questions.  Factor 
analysis  is  a  technique  used  to  identify  and  summarize  the 
inter-relationsu  us  that  exists  among  the  individual 
questions.  For  this  thesis,  factor  analysis  will  be  used  to 
confirm  whether  or  not  there  is  a  correlation  between  the 
perceived  worst  problems  and  the  top  ranked  suggested 
improvements.  Factor  analysis  is  a  higher  order  data 
reduction  technique  that  can  systematically  reduce  the  large 
correlation  matrices  resulting  from  correlating  all  of  the 
survey  questions  (19:378).  This  helps  identify  a  grouping 
of  the  questions  that  those  surveyed  answered  in  a  similar 
manner.  Since  factor  analysis  identifies  groupings  of  the 
survey  questions  that  are  highly  correlated  with  each  other 
(19:380),  the  factor  analysis  can  be  used  to  confirm  whether 
or  not  the  logical  groupings  have  identified  correlated 
survey  responses . 

Computer  Tabulation.  Statistical  analysis  is  done 
using  SPSS-X  (26)  and  the  personal  computer  (PC)  software 
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package  called  STATISTIX  (25).  SPSS  is  an  integrated  set  of 
computer  programs  that  are  capable  of  doing  correlations, 
nonparametric  tests  and  reliability  analysis.  The  data  from 
the  Teal  Survey  is  analyzed  using  SPSS-X  and  STATISTIX.  For 
performing  repeated  calculations,  the  PC  software  package 
Quattro  (4)  was  used. 
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Ill .  A  Synopsis  of  Studies  Involving 
Data  and  Data  Management 

This  chapter  contains  a  collection  of  unpublished 
studies  conducted  by  various  agencies  within  AFSC.  Each 
study  is  synopsized  and  a  summary  of  commonalties  found 
across  the  studies  is  included.  Each  study  was  initiated 
because  of  a  perception  that  the  acquisition  process  was  too 
cumbersome.  The  studies  were  initiated  to  search  for  ways 
to  improve  the  process.  Since  some  of  the  studies  cited 
here  included  areas  other  than  data  and  data  management, 
this  discussion  will  include  only  those  areas  related  to 
data  . 

Electronic  Systems  Division,  (ESP) 

In  August  1981,  the  final  report  of  ESD's  Data  Review 
Action  Task  (DRAT)  was  delivered.  DRAT  was  initiated  by  the 
ESD  Commander,  Lt  Gen  Robert  T.  Marsh  and  completed  under 
Lt  Gen  James  W.  Stansberry.  DRAT's  purpose  was  to  conduct  a 
study  to  assess  the  perceived  problem  of  the  government 
ordering  too  much  data  on  its  contracts.  The  goal  of  the 
DRAT  study  was  to  recommend  appropriate  actions  as  required 
and  establish  controls  for  continued  management  of  future 
data  acquisitions  (12).  The  following  is  a  synopsis  of  the 
DRAT  team's  findings: 

1.  The  government  orders  too  much  data  in  requests  for 
proposals  (RFPs)  and  contracts.  To  reduce  the  amount  of 
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data  requested,  the  DRAT  recommended  the  development  of  an 
ESD  Data  Management  Guide  to  better  focus  the  requests  for 
data.  The  DRAT  study  also  advocated  assigning  a  "journeyman 
level"  data  management  officer  (DMO)  to  each  program  office, 
plus  establishing  a  Senior  ESD  Data  Requirements  Review 
Board  (DRRB)  that  would  remove  the  responsibility  for 
accomplishing  the  DRRB  function  from  the  project  officer.* 
While  elevating  the  level  of  the  data  manager  in  a  program 
office  would  help  eliminate  duplicate  efforts  by  government 
personnel,  the  DRAT  emphasized  the  need  for  more  senior 
level  involvement  in  the  data  management  process  (12). 

2.  Acquisition  and  processing  of  contractor  data 
consumes  a  large  share  of  ESD  resources  (money  and  peoplej. 
Since  neither  the  actual  cost  of  data  nor  the  number  of  man¬ 
hours  expended  on  data  management  has  been  determined,  how 
large  of  a  share  is  unknown.  To  control  the  amount  of 
government  resources  expended  to  manage  data,  the  DRAT  study 
recommended  establishing  programs  to  increase  awareness  that 
data  is  a  cost  driver  (12). 

3.  The  DRAT  study  found  that  data  is  a  significant 
cost  driver  that  has  been  neglected  by  senior  level 
managers.  Lack  of  consistent  data  pricing  rules  make  it 


*  By  1989,  ESD  held  two  DRRBs .  One  at  the  program 
office  level,  and  a  second  by  senior  management.  The  goal 
of  the  second  was  to  reduce  the  data  requests  from  the  first 
DRRB  by  10-20%.  Also,  ESD  was  developing  the  "preferred 
data  list  of  96  DIDs  needing  no  justification,  only 
tailoring,  to  fit  the  program  (15). 
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impossible  to  accurately  record  and  track  data  costs. 
However,  the  Office  of  Federal  Procurement  Policy  has 
estimated  annual  costs  of  data  on  all  procurements  to  be 
several  billion  dollars.  ESD  estimates  data  costs  range 
from  6-20%  of  contract  costs.  However,  the  ESD  data  cost 
estimate  does  not  include  the  actual  proposal  preparation 
costs  (partially  recovered  only  upon  contract  award  as  bid 
and  proposal  costs)  or  other  costs  which  become  part  of  a 
contractor's  overhead  and  thereby  subsequently  are  charged 
to  the  government.  Nor  does  the  ESD  estimate  include  the 
cost  to  the  government  for  government  people  in  processing, 
reviewing  and  managing  the  data.  While  the  DRAT's  Final 
Report  does  not  lead  the  reader  to  believe  the  cost  of  data 
and  data  management  will  ever  be  known  exactly,  it  does 
suggest  that  higher  level  attention  to  the  quantity  and 
quality  of  data  could  significantly  reduce  costs.  The  DRAT 
Report  suggests  that  the  data  manager  be  made  the  focal 
point  for  all  organization  data  related  activities,  and 
suggests  the  establishment  of  participative  management  so 
data  managers,  program  managers  and  data  users  play  a 
greater  role  in  reducing  data  requirements.  The  DRAT  Report 
also  suggests  that  program  managers  be  required  to  report 
data  costs  in  Program  Management  Reviews.  Finally,  the  DRAT 
Report  recommends  that  separate  prices  for  data  be  included 
in  the  contract  schedule,  and  that  the  government  provide 
additional  guidance  to  the  contractor  on  what  specific 
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efforts  should  be  included  in  estimating  and  revealing 
actual  data  prices  (12). 

4.  Data  management  suffers  from  a  lack  of  experienced 
managers  and  from  inadequate  data  manager  training,  both  on- 
the-job  and  in  formal  training  courses.  The  DRAT  suggested 
several  aspects  of  data  manager  training,  including  a 
training  program  with  industry  and  an  AFIT  continuing 
education  course  (12).  (The  DRAT  Report  does  not  elaborate 
on  whether  its  authors  are  suggesting  the  development  of  a 
new  course,  or  merely  referring  to  the  AFIT  Sys  370,  Defense 
Data  Management  course,  already  offered.) 

5.  Data  management  pricing  concepts  have  not  been 
consistently  applied,  verified,  nor  archieved  for  automated 
retrieval,  thus  making  any  attempts  for  use  of  historical 
data  for  trend  analysis  or  price  analysis  impractical .  The 
DRAT  recommends  ESD  devise  an  automated  central  data  pricing 
system  for  recording  and  analyzing  data  prices  (12). 

In  summary,  the  DRAT  Report  repeatedly  stressed  the 
need  for  higher  level  manager  involvement  in  the  data  and 
data  management  process.  The  DRAT  Report  concluded  that 
data  was  a  large  cost  driver  which,  at  the  time  of  the 
study,  had  received  little  attention  from  upper  level 
management.  The  DRAT  Report  recommended  higher  level 
management  participation  in  14  of  their  83  recommendations. 
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Aeronautical  Equipment  ( AE)  System  Program  Office  ( SPQ) 


On  May  4,  1989,  the  ASD/AE  SPO  completed  its  Critical 
Process  Team  Final  Report  on  the  AE  purchase  request 
process.  As  part  of  an  effort  to  implement  the  Total 
Quality  Management  (TQM)  philosophy,  the  executive  level  of 
the  AE  SPO  identified  the  purchase  request  (PR)  process  as 
one  of  the  most  "painful"  processes  within  AE,  and  an  area 
where  improvements  could  lead  to  enhanced  mission 
accomplishment.  AE  established  a  critical  process  team 
(OPT)  to  investigate  and  evaluate  the  PR  process  and 
implement  improvements  through  the  SPO  Director.  The  CPT ' s 
goal  was  to  eliminate  non-essential  elements  and  optimize 
essential  elements  of  the  PR  process.  The  following  are 
problems  with  the  PR  process  that  the  CPT  found. 

1.  There  was  no  documentation  directing  the  PR 
process.  Discrete  elements  of  the  process  were  put  forth  in 
various  instructions  (i.e.  AEOI  20-1,  AEOI  57-2.  and 

AEOI  310-1),  but  there  was  no  single  source  that  gave  an 
overall  perspective  of  the  process  from  beginning  to  end. 

The  CPT  recommended  assembly  of  a  handbook  documenting  the 
PR  process  (21:3) . 

2.  A  lack  of  early  and  complete  involvement  by 
engineering  personnel  due  to  time  constraints  leads  to 
poorly  defined  technical  requirements.  This  has  two 
simultaneous  negative  effects:  a.  The  wrong  or  incomplete 
requirements  are  specified,  and  b.  Costly  changes  must  be 
made  later  in  the  program  to  correct  inadequate 
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requirements.  The  CPT  recommended  early  engineering 
involvement  and  an  adequate  amount  of  time  be  made  available 
for  thorough  engineering  analysis  and  requirements 
definition  (21:5) . 

3.  Participants,  especially  non-SPO  participants, 
required  for  the  data  call  process  to  be  complete  are 
typically  non-responsive  to  the  original  data  call.  This 
results  in  significant  time  delays  in  the  PR  processing 
cycle  to  follow  up  with  these  offices  to  receive  their 
input.  The  CPT  recommended  developing  a  "standard  data 
package"  for  use  by  all  project  teams.  This  "standard  data 
package"  would  include  data  which  is  common  to  virtually  all 
AE  programs.  This  would  save  time  at  the  Data  Requirements 
Review  Board  (DRRB)  and  eliminate  the  need  for  Air  Force 
Form  585s  (justifications)  for  those  data  items  in  the 
package.  This  would  then  allow  the  DRRB  to  concentrate  on 
only  those  items  that  are  program  tailored  (21:10). 

4.  Due  to  the  heavy  involvement  of  inexperienced 
personnel  at  the  DRRB,  the  quality  of  the  CDRL  package 
developed  is  often  less  than  desired.  The  CPT  recommended 
that  each  organization  participating  in  the  PR  process  be 
required  to  designate  one  primary  and  one  alternate  DRRB 
member  (21:10).  The  implication  here  is  that  consistency  of 
personnel  involved  in  the  DRRB  will  lead  to  an  improved 
DRRB.  The  final  report  made  no  other  mention  of  specific 
training  requirements  or  experience  levels  recommended  for 
DRRB  members . 
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Ballistic  Systems  Division  (BSD)  Study  on  Statement  of  Work 
( SOW)  Preparation  Process 

This  study  performed  at  BSD  and  completed  in  December 
1989  was  initiated  to  investigate  the  SOW  preparation 
process  for  the  Small  Intercontinental  Ballistic  Missile 
(ICBM)  program.  The  Small  ICBM  program  was  in  the  Full 
Scale  Development  phase,  and  government  and  contractor  input 
was  solicited  for  this  study.  The  following  recommendations 
pertaining  to  data  and  data  management  were  included  in  the 
study : 

1.  The  study  recommended  establishing  a  SOW/CDRL 
requirements  review  board  consisting  of  2-Letter  and  3- 
Letter  chiefs  that  make  up  the  program  management  team 
(PMT) .  This  presents  a  high  level  management  emphasis  to 
ensure  the  SOW/CDRL  contains  only  minimum  essential 
requirements  and  is  within  budget  constraints  (17). 

2.  Minimize  the  total  number  of  CDRLs  ordered  by 
eliminating  non-essential  reports,  reducing  multiple  copy 
requirements,  and  minimizing  the  drafts  for  approval.  This 
requires  the  thorough  screening  of  CDRL  requirements  by 
personnel  trained  and  knowledgeable  about  the  process,  and 
implies  the  necessity  of  high  level  program  management 
involvement  to  "make"  these  actions  happen  (17). 

3.  Contractor  input,  with  regard  to  data,  cited  that 
increasing  the  amount  of  downward  tailoring  of 
specifications  and  DIDs  placed  on  CDRLs  would  reduce  data 
costs  to  the  government.  The  contractor  recommended  a 


33 


"zero-based"  approach  to  tailoring,  which  means  starting 
from  scratch  and  defining  needs,  instead  of  using  previous 
contracts  as  examples  or  proliferating  a  list  of  possible 
DIDs .  The  contractor  recommendation  also  included  a  call 
for  more  contractor  input  to  the  draft  RFP,  SOW,  and  CDRL. 
This  would  provide  increased  opportunity  for  clarification 
of  requirements  prior  to  going  on  contract.  These  views 
were  echoed  by  all  12  contractors  responding  to  BSD's 
request  for  their  input,  as  well  as  two  AFPROs ,  Air  Force 
Plant  Representative  Offices  (17). 

Training  Systems  ( YW)  System  Program  Office 

In  October  1987,  ASD/YW  established  a  critical  process 
team  (CPT)  charged  with  examining  the  RFP  process,  finding 
ways  to  streamline  the  process,  and  creating  tools  to  ensure 
each  event  of  the  process  "gets  done  right  the  first  time." 
The  CPT  used  an  interview  questionnaire  to  gather  input  from 
individuals  involved  in  previous  RFP  preparations.  Listed 
below  are  those  areas  relating  to  data  and  data  management 
that  the  CPT  found  to  be  problems  or  burdensome  to  the  RFP 
process . 

1.  When  developing  the  requirements  for  the  RFP.  there 
is  a  tendency  for  non-user  (those  not  in  the  using  command) 
requirements  to  creep  into  the  RFP  during  the  coordination 
process.  This  usually  occurs  when  persons  other  than  the 
user  add  technical  enhancements,  quality  programs,  and 
reliability  demonstrations  to  the  requirements.  Also, 
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contract  options  are  often  added  to  a  contract  to  hedge 
against  funding  cuts  and  uncertain  or  changing  requirements. 
These  actions  add  unnecessary  work  to  RFP  preparation,  add 
to  the  amount  of  data  requested  but  not  used,  and  add 
unnecessary  requirements  and  ambiguities  to  the  contract 
(27:5)  . 

2.  A  common  misperception  is  that  the  program  manager 
is  an  expert  in  RFP  preparation  and  statement  of  work  (SOW) 
writing,  when  in  fact  many  of  YW  program  managers  had  never 
written  a  SOW  before.  This  reveals  the  lack  of  training  and 
experience  in  RFP  and  SOW  preparation  that  is  common  even 
among  senior  level  management  (27:6). 

3.  The  data  call  process  is  flawed  because  the  using 
command  and  AFLC  have  not  typically  seen  the  SOW  prior  to 
the  data  call.  This  turns  the  data  call  into  an  educational 
process,  addressing  progress  and  milestones  schedules  of  the 
program,  rather  than  concentrating  on  the  purposes  of  a  data 
call  and  DRRB.  The  result  is  a  poorly  conducted  DRRB  that 
does  not  address  the  issue  of  reviewing  data  requests  for 
appropriateness.  The  end  result  is  often  a  contract 
containing  duplicate  and  unnecessary  data  requests  (27:7). 

4.  The  CPT  cited  conflicting  direction  from 
regulations  as  a  problem  in  preparing  RFPs .  The  CPT  cites 
conflicting  DoD  directives  and  instructions,  FAR  and  FAR 
supplements,  agency  regulations  and  organizational 
instructions  as  major  impediments  to  the  RFP  process. 
Examples  the  CPT  cites  are  listed  below. 
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a.  "Acquisition  Streamlining"  requires/recommends 
that  program  management  tailor  military  standards  and 
specifications  for  a  specific  program.  However,  AFSCR 
800-9  discourages  tailoring  MIL-STD-1567A  and  any 
changes  must  be  approved  by  ASD/CC.  Also,  AFR  800-9 
requires  application  of  MIL-STD-1567A  to  programs 
except  for  competitive  firm  fixed  price  contracts,  but 
AFSCR  800-9  does  not  allow  this  exception  (27:20). 

b.  FAR  Part  7  and  supplements  identify  the 
contracting  officer  as  being  responsible  for 
Acquisition  Plan  preparation.  However,  ASD/PM  policy 
for  contracting  personnel  collocated  in  a  program 
office  requires  program  management  to  write  the  plan 
(27:20) . 

c.  Streamlining  initiatives  encourage  the  use  of 
contractor  format  and  accepted  commercial  practices 
while  the  ASD  review  process  enforces  the  requirements 
as  set  forth  under  the  MIL-STDs  (27:21). 

5.  The  experience  level  of  those  involved  in  the  RFP 
process  was  identified  as  a  major  weakness.  The  CPT  found 
that  86%  of  those  involved  in  the  process  had  less  than  2 
years  SPO  experience  and  had  only  an  introductory  course  in 
acquisition.  Only  7%  of  those  involved  had  over  8  years  of 
experience  and  advanced  level  training.  This  lack  of 
experience  was  complicated  further  by  the  20%  turnover  of 
personnel  every  six  months  in  YW.  The  CPT  concluded  that 
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the  inexperience  and  turnover  of  personnel  greatly  impacted 
the  quality  of  RFPs  (27:25). 

Data  Requirements  Streaml ininq:  Engineering  Paper  Reduction 
Study 

In  1989,  ASD/EN  performed  a  study  to  determine  where 
the  engineering  effort  in  systems  acquisition  could 
streamline  the  process  and  reduce  the  amount  of  paperwork. 
The  study  included  contractor  input,  analysis  of  data  on 
selected  contracts,  and  interviews  with  data  managers.  The 
goals  of  the  study  were  to  characterize  the  deliverable 
data,  drive  deliverable  data  toward  zero,  and  provide 
leadership  for  other  ASD  organizations  in  their  effort  to 
streamline  data  requirements.  The  following  is  a  summary  of 
the  study  group’s  findings. 

1.  The  study  group  interviewed  contractors  for  their 
views  on  reducing  the  data  requirements.  Contractors 
revealed  that  the  government  efforts  to  price  data  would  be 
difficult  because  the  government  requires  an  artificial 
separating  out  of  data  prices  from  work  task  prices,  and 
contractors  do  not  have  this  information  easily  available. 
Also,  contractors  have  the  perceptions  that  the  government 
does  not  need  or  read  the  reports  it  requests.  Contractors 
believed  the  government  should  reduce  the  amount  of 
requested  data,  the  frequency  of  data  submittals,  the  number 
of  copies,  and  the  wide  distribution.  Contractors  suggested 
the  government  use  contractor  formats,  request  fewer  plans. 
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avoid  duplicate  requests,  and  recognize  that  technical 
manuals  and  software  data  are  high  dollar  cost  drivers  (2). 

2.  The  study  group  examined  the  CDRL  items  used  by 
ASD/EN  and  assigned  them  into  three  categories.  The 
categories  were:  a.  permanent  -  which  meant  the  information 
was  needed  for  future  government  use,  b.  perishable  -  which 
meant  the  information  was  needed  only  one  time,  and 

c.  historical  -  which  meant  the  information  might  possibly 
be  useful  in  the  future.  The  group  found  that  of  the 
engineering  data  requested,  33%  were  for  permanent  data.  36% 
were  for  perishable  data,  and  31%  were  for  historical 
data  ( 2 ) . 

3.  The  study  group  reviewed  the  data  management 
process  and  found  that  while  AFR  310-1,  Management  of 
Contractor  Data,  provided  for  a  DRRB  to  minimize  data 
requests,  data  requesters  anticipate  DRRB  resistance.  (The 
report  implied  that  this  anticipation  of  resistance  impeded 
the  effort  to  reduce  data.)  The  group  found  little 
incentive  for  functional  levels  to  reduce  data  requests. 

The  study  group  suggested  that  better  training  on  data 
management  at  the  functional  level  would  enable  a  better 
screening  of  data  requests  than  is  currently  done  by  the 
DRRB  ( 2 ) . 

4.  The  study  group  asked  the  "big  question,"  i.e.  What 
essential  data  do  you  need  to  be  formally  delivered  to  you 
in  order  for  you  to  do  your  job?  The  responses  were  (2): 
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to  maintain 


a.  Delivered  data  is  not  required 
insight  into  program  technical  status. 

b.  Delivered  data  is  not  required  for  quick 
reactions  to  analyzing  and  resolving  technical 
problems . 

c.  Special  data  requests  may  be  necessary  to 
conduct  independent  assessments. 

d.  Delivered  data  is  necessary  for  historical 
files  for  future  reference. 

e.  The  group  found  that  scrubbing  their  data 
requests  enabled  them  to  reduce  the  number  of  data  item 
descriptions  (DIDs)  from  225  to  66. 

5.  ASD/EN  concluded  that  a  key  ingredient  to  reducing 
data  was  involvement  by  top  level  management.  They  found 
data  can  be  reduced,  and  the  primary  responsibility  to  do  so 
is  first  placed  with  the  functional  areas  in  pre-scrubbing 
data  orders  before  going  to  the  DRRB.  They  felt  this  could 
only  be  done  if  data  management  training  were  to  emphasized 
at  the  functional  level  (2). 

Munition  Systems  Division  (MSP)  Acquisition  Process 
Improvement  Team 

In  December  1989,  the  MSD  Acquisition  Process 
Improvement  Team  (MAPIT)  presented  their  final  report. 

MAPIT  conducted  a  study  designed  to  gather  information  on 
problems  occurring  during  the  acquisition  process  and 
determined  steps  that  could  be  taken  to  improve  the  process. 
Input  to  the  study  consisted  of  interviews  with  those 
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individuals  involved  in  the  process  (both  government  and 
contractor)  and  analysis  of  policies  and  procedures.  While 
MAPIT  investigated  several  areas  of  the  process,  the 
following  discussion  is  limited  to  their  findings  concerning 
data . 


1.  From  their  study,  MAPIT  concluded  that  the  current 
practice  of  having  a  SOW/specification  data  call  should  be 
eliminated.  Instead,  MAPIT  recommended  conducting  in  a 
concurrent  manner  the  SOW,  specification,  and  CDRL 
developments,  culminating  with  a  system  requirements  call. 
This  would  then  alleviate  the  duplicity  and  ambiguities 
inherent  with  the  current  data  call  process.  MAPIT  also 
recommended  establishing  standard  data  packages,  thus 
allowing  review  boards  more  time  to  evaluate  the  need  for 
additional  data  requests.  (The  standard  data  packages  would 
not  be  subject  to  DRRB  scrutiny.)  If  these  steps  were 
implemented,  MAPIT  estimated  the  time  from  SPO  resource 
commitment  to  contract  award  could  be  reduced  by  13  weeks 
from  the  current  109  weeks  and  improve  the  quality  of  the 
contract  ( 23) . 


2.  MAPIT  recommended 
Form  1423  (the  CDRL  form), 
the  relative  importance  of 
relative  to  it's  cost,  the 
price  categories  be  listed 


the  pricing  of  data  items  on  DD 
To  assist  management  in  deciding 
the  requested  information 
team  suggested  the  following 
on  each  DD  Form  1423  (23) : 
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a.  Development  costs:  (Those  costs  incurred  if 
the  data  is  not  generated  as  a  part  of  normal  business, 
but  is  requested  by  the  government.) 

b.  Technical  preparation  costs:  (Those  costs 
incurred  by  the  contractor  to  place  the  data  into  a 
usable  format.  By  accepting  contractor  format,  this 
cost  can  be  minimized.) 

c.  Administrative  preparation:  (Those  normal 

administrative  costs  associated  with  document 
assembly . ) 

d.  Reproduction  and  delivery:  (Normally 

negligible  costs.) 

3.  To  reduce  the  amount  of  data  requested,  an  MSD 
"minimum  essential  data  list"  was  recommended.  This  list, 
developed,  scrutinized,  and  agreed  to  by  the  functional 
disciplines,  would  require  no  further  justification.  Time 
previously  spent  on  justifying  items  on  the  minimum 
essential  data  list  could  be  used  to  scrutinize  the 
tailoring  and  distribution  of  data  requirements  and  in 
justifying  the  remaining  data  items  focused  to  the 
particular  program.  In  addition  to  this  minimum  essential 
data  list,  the  study  group  developed  a  set  of  conditions, 
which  when  met.  would  permit  the  addition  of  other  data 
items  to  the  minimum  list  (23). 

4.  The  study  group  recommended  automation  of  the 
acquisition  functions  be  made  a  top  priority  of  MSD.  In 
particular,  the  Automated  CDRL  and  Tracking  System  was 
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recommended.  Automation  was  suggested  to  partially 
alleviate  problems  from  reductions  in  manpower,  <*.r  w<* 1  ’  js 
enhance  the  quality  of  the  CDRLs  and  tracking  of  deliverable 
data  ( 23 ) . 

5.  Unclear,  ambiguous,  unquanti f lable ,  and  excessive 
data  requirements  were  among  the  problems  the  group  found. 
The  reasons  they  cited  for  this  included  inexperienced  and 
poorly  trained  personnel.  Solutions  they  recommended  were 
increasing  the  involvement  of  senior  management  and 
experienced  people,  providing  additional  training,  and 
developing  a  bipartisan  RFP  review  process  (23). 

Summary  of  Unpubl ished  Studies 

The  following  is  a  summary  of  the  previous  six  studies. 
Since  these  studies  are  not  exhaustive  nor  necessarily 
directed  toward  the  same  end.  there  is  no  attempt  made 
herein  to  statistically  analyze  their  findings.  However, 
the  studies  do  reveal  similar  problem  trend  areas  and 
overlap  coverage  in  the  recommended  corrective  actions 
thereby  transcending  any  investigative  differences  the 
individual  studies  possess.  These  significant  problem  trend 
areas  and  recommended  corrective  actions  evident  in  more 
than  one  study  are  summarized  below. 

a.  Highest  Significant  Trend  -  The  government 
asks  for  too  much  data.  This  problem  was  cited  by  five 
of  the  six  studies. 
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b.  Highest  Significant  Corrective  Action  -  Five 
studies  cited  the  need  for  more  training  of  personnel 
involved  in  the  acquisition  process.  Both  the 
inexperienced  and  the  "experienced"  were  found  to  be  in 
need  of  training  on  RFP,  SOW,  and  specification 
preparation,  as  well  as  data  tailoring. 

c.  Second  Highest  Corrective  Action  -  Four 
studies  cited  the  need  for  more  involvement  by  senior 
level  personnel  (i.e.  the  program  manager)  in  the 
acquisition  process.  The  purpose  of  this  was  to 
elevate  the  perception  of  importance,  but  more 
importantly  to  get  senior  level  management  to  recognize 
the  importance  of  the  document  preparation  process  and 
guide  the  evolution. 

d.  Three  of  the  study  groups  cited  the  importance 
of  pricing  data  items.  It  was  acknowledged  that 
pricing  data  was  not  easy,  and  would  never  be  100?& 
accurate,  but  the  effort  offered  rewards  both  in 
determining  the  real  need  for  the  data  and  in  improving 
the  quality  of  the  data  requirements. 

e.  Two  studies  cited  data  as  a  contract  cost 
driver . 

f.  The  benefits  of  determining  a  standard  data 
package  that  avoided  the  data  justification  phase  was 
emphasized  in  two  of  the  studies. 

g.  A  recommendation  for  overhauling  the  data  call 
and  DRRB  process  was  made  by  two  of  the  study  groups. 
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h.  ESD  and  MSD  called  for  automating  the 
acquisition  process  as  much  as  possible.  In 
particular,  CDRL  preparation  and  tracking  were 
identified  as  areas  to  automate.  (ESD  and  MSD  to  a 
large  extent  have  automated  these  parts  of  the 
acquisition  process.  The  success  they  have  experienced 
is  inferred  to  be  a  reason  for  their  endorsement  of 
automation . ) 

The  POD  Career  Series  Survey 

This  survey,  performed  in  the  fall  of  1989,  has  become 
known  as  the  DM/CM/EDM  survey  because  of  its  emphasis  on 
data  managers,  configuration  managers,  and  engineering  data 
managers.  A  summary  of  this  survey  is  included  in  this 
thesis  because  it  not  only  reveals  who  works  with  data,  but 
shows  how  any  corrective  actions  to  the  current  problems 
will  be  difficult  to  implement  due  to  the  lack  of  an  easily 
identifiable  work  force.  The  DM/CM/EDM  survey  was  initiated 
to  facilitate  the  DOD  effort  to  create  an  acquisition  work 
force.  The  survey  was  directed  at  the  civilian  personnel 
since  the  researchers  task  was  limited  to  determining  the 
role  of  civilians  in  the  acquisition  work  force  and  the 
belief  that  civilians  perform  most  of  the  DM,  CM,  and  EDM 
functions.  The  survey  provided  information  on  the  numbers 
of  people  performing  the  work  of  a  DM,  CM,  and  EDM  in  the 
DOD,  since  there  is  no  single  career  series  designated  to 
perform  these  functions.  It  was  important  to  determine  who 
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really  manages  data  In  order  to  know  who  to  train  and/or  how 
to  manage  this  work  force,  especially  as  the  DOD  transitions 
into  digitized  data  deliveries  (14). 

In  the  Air  Force,  there  are, 32  civilian  career  series 
that  perform  DM,  CM,  and/or  EDM  functions.  Of  the  2418  Air 
Force  respondees,  1391  perform  this  work  on  a  part  time 
basis,  while  1027  are  full  time.  However,  a  single  career 
series,  the  GS-1670  Equipment  Specialist,  accounts  for  1053 
of  persons  performing  CM  on  a  part  time  basis  in  AFLC .  The 
break  out  for  DM.  CM,  and  EDM  by  military  services  is  as 
f ol lows  ( 14) . 


TABLE  1 

Number  of  DM/CM/EDM  Working  Full  or  Part  Time 

by  Service  (14) 


Air  Force 

Navy 

Army 

Total 

Full  Time 

1027 

453 

226 

1706 

Part  Time 

1391 

53 

361 

1805 

Sub-total 

2418 

506 

587 

3511 

The  survey  found  that  it  was  common  for  a  person  to  perform 
the  functions  of  a  DM,  CM,  and  an  EDM  simultaneously  (14). 

The  survey  results  revealed  some  significant 
differences  between  the  way  the  Navy  and  the  Air  Force 
approach  data  management.  The  Navy  uses  53  part-time  people 
compared  to  the  Air  Force  with  1,391  part-time  people.  This 
could  be  because  the  Navy  has  a  certification  program  for 
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its  data  managers.  (Naval  Sea  Systems  Command  for  DMs  and 
Navy  Supply  Command  for  EDMs.)  They  also  have  assigned  for 
several  years  centralized  permanent  members  for  their  DRRBs 
on  higher  dollar  programs.  By  contrast,  the  Air  Force  has 
ad  hoc  DRRBs  with  few  permanent  members  and  thereby  uses 
more  people  than  the  other  services.  This  could  be  because 
the  Air  Force  does  not  elevate  the  level  of  importance  of 
data  managers  as  the  Navy  has  done.  This  inefficiency  in 
the  Air  Force  use  of  its  people  will  be  difficult  to  defend 
in  the  restructuring  now  being  mandated  by  Secretary  Cheney 
(15).  The  Air  Force  recruits  data  managers  primarily  out 
of  clerical  jobs  in  grades  GS-3,  GS-4 ,  and  GS-5  (civilian 
career  series  303  which  tops  at  GS-6) ,  converting  them  to 
general  administrative  (career  series  301  which  tops  at  GM- 
13)  where  promotions  are  available  (14). 

From  the  information  available  from  the  DM/CM/EDM 
survey,  it  appears  the  lack  of  recognizing  the  data  manager 
function  as  an  important  acquisition  work  force  career 
position  could  be  having  very  costly  negative  effects  on  the 
Air  Force  acquisition  process.  Future  research  may  be 
warranted  to  compare  Navy  versus  Air  Force  contract  data 
costs.  If  a  correlation  could  be  drawn  between  the  grade, 
experience,  authority,  and  training  level  of  data  managers 
in  effectively  reducing  data  costs,  the  Air  Force  could 
potentially  reduce  acquisition  costs  significantly  by 
selectively  adopting  the  Navy's  approaches  to  data 
management  in  selecting,  training,  and  certifying 
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(empowering)  data  managers.  Also,  the  functions  and  roles 
of  support  contractors  should  be  studied  to  see  how  the  Navy 
and  Air  Force  utilize  these  specialists  in  managing  data  in 
the  acquisition  and  logistics  arenas. 
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IV .  Results  of  Auditor  Report  and  Tea  1  Survey  Analysis 


This  chapter  is  devoted  to  analyzing  the  Auditor 
General's  report  and  the  Teal  Survey.  A  summary  of  each  is 
included,  though  commonalties  and  integrated  observations 
are  left  for  Chapter  Five.  Descriptions  of  the  analysis 
techniques  are  included  in  Chapter  Two. 

The  Auditor  General ' s  Report ,  Project  No .  6066416 

In  1988,  the  Office  of  the  Auditor  General  of  the  Air 
Force  evaluated  Air  Force  policies  and  procedures  for 
acquiring  and  managing  data  during  system  acquisition.  For 
this  study,  data  is  defined  to  include  administrative, 
management,  financial,  technical,  scientific,  engineering, 
and  logistics  information  required  to  manage  a  contractual 
effort.  (This  definition  of  data  is  consistent  with  the 
concept  of  data  used  thus  far.)  A  total  of  32  Air  Force 
contracts  totaling  over  $10  billion  were  examined  for  this 
report.  The  auditor's  report  concentrates  on  three  areas: 
1.  Unessential  and  uneconomical^  acquired  data,  2.  Data 
tracking,  and  3.  Data  pricing.  This  thesis  will  analyze 
only  the  unessential  and  uneconomi ca 1 ly  acquired  data 
portion  of  the  auditor's  report.  The  auditor's 
interpretation  of  the  data  is  included  as  well  as  an 
alternate  approach  to  interpreting  the  data  (3:1). 
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The  Auditor  Genera  I ' s  View.  Twenty-eight  AFSC 
contracts  were  reviewed,  and  data  were  collected  on  each 
contract's  total  dollar  value  along  with  a  measure  of 
dollars  estimated  to  be  in  the  data  costs  on  each  contract. 
Contract  data  items  were  specifically  investigated,  with 
emphasis  on  the  justification  of  the  need  for  the  data  and 
the  appropriateness  of  the  method  of  delivery.  To 
accomplish  the  objective  of  determining  the  magnitude  of  the 
problem  of  unessential  and  uneconomi ca 1 ly  acquired  data,  32 
contracts  from  MSD,  ASD,  BSD,  ESD .  and  SSD  were  selected. 

The  only  criteria  for  contract  selection  were  that  the 
contract  value  had  to  exceed  $1  million,  contain  at  least  -0 
data  items,  and  be  for  research/development  or 
procurement/production.  While  the  contract  dollar  amount  of 
these  contracts  was  known,  some  of  the  data  costs  could  only 
be  estimated  due  to  contracting  officers  not  breaking  the 
data  prices  out  from  a  negotiated  contract  price  (3:4). 

With  the  contract  and  data  costs  established,  a 
representative  sample  of  data  items  was  chosen  from  each 
contract.  (The  assumption  was  made  that  the  data  in  the 
sample  were  representative.  While  the  auditor's  report  was 
not  explicit  about  this,  there  was  no  mention  of  limiting 
the  sample  to  specific  data  items.  In  fact,  limiting  those 
data  items  eligible  for  the  sample  could  yield  misleading 
inferences.  In  looking  at  the  wide  range  of  sample  data 
prices  and  the  number  of  data  items  from  each  contract,  the 
assumption  of  a  representative  sample  seemed  to  be 
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reasonable.)  Those  data  items  in  the  sample  were  then 
examined  for  unessential  and  uneconomical ly  acquired  data. 

The  analysis  of  the  sample  data  only  showed  that  61 
of  411  data  items,  at  a  total  cost  of  $2.6  million,  were 
unessential  and  21  of  the  remaining  350  data  items,  at  a 
total  cost  of  #2.6  million,  were  uneconomica  1 ly  acquired. 
(Data  items  found  to  be  unessential  were  dropped  from 
further  scrutiny.)  From  this  information,  the  auditor's 
report  concluded  that  out  of  $10  billion  worth  of  total 
contract  dollars,  $5.2  million  was  spent  on  unessential  or 
uneconomical ly  acquired  data  (3:1).  This  is  a  misleading 
conclusion,  since  the  $5.2  million  of  unessential  and 
uneconomi ca 1 ly  acquired  data  was  from  only  the  sample.  The 
alternative  analysis  of  this  data  will  address  this  issue. 
The  possibly  more  important  conclusions  from  the  report  as 
to  why  the  problem  of  unessential  and  uneconomical ly 
acquired  data  exists  are  worth  noting  and  are  listed  below. 

a.  Data  calls  were  not  issued  or  even  when  issued, 
adequate  time  was  not  allowed  for  responses.  Complete 
information  was  not  provided  nor  could  the  auditors  retrieve 
enough  information  for  many  of  the  contracts,  however  the 
information  below  is  a  summary  of  the  auditor’s  findings. 

1)  Of  26  contracts  for  which  information  was 
available,  21  (80%)  held  data  calls  (3:7). 

2)  Of  21  contracts  that  had  data  calls  issued, 
information  was  available  for  18  contracts  on  the  time 
the  allowed  for  responses  to  the  data  calls.  Times 
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allowed  ranged  from  six  days  on  two  contracts  to  42 
days  on  one  contract,  with  the  average  being  21  days. 
Eleven  contracts  allowed  less  than  30  days  response 
time  despite  AFR  310-1  requirement  for  30  days  (3:7-8). 

3)  Even  though  21  of  the  contracts  issued  data 
calls,  only  15  DRRBs  were  convened  and  of  these  only 
three  were  ruled  effective  by  the  auditors  (3:10). 

4)  None  of  the  28  contracts  performed  an  economic 
analysis  on  the  requested  data  even  though  AFR  310-1. 
Management  of  Contractor  Data,  requires  an  economic 
justification  (3:14). 

b.  The  Auditor's  report  cites  several  instances  where 
their  analysis  had  to  be  curtailed  because  of  a  lack  of 
information.  The  missing  information  was  usually  a  lack  of 
cost  information.  For  example,  of  the  411  data  items 
examined  in  the  sample,  the  auditor  could  only  identify 
prices  for  319  items  (3:7).  This  inability  to  determine  the 
actual  cost  of  deliverable  data  items  leaves  program 
managers,  contract  officers,  and  auditors  vulnerable  to 
spending  the  taxpayer's  dollar  in  a  less  than  prudent 
manner . 

An  Alternate  Ana  lysis .  Of  the  28  contracts  reviewed 
for  unessential  and  uneconomica  1 ly  acquired  data, 
information  on  data  costs  were  available  on  only  26 
contracts.  Therefore,  only  26  contracts  should  be  used  for 
analytical  purposes  and  conclusions.  These  26  contracts  had 
a  total  value  of  $5.8  billion  and  a  total  of  2819  data  items 
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with  an  estimated  value  of  $95  million.  The  auditors  took  a 
sample  of  data  items  from  each  contract.  The  method  used  to 
determine  how  many  data  items  from  each  contract  would  be 
included  in  the  sample  was  not  explained  and  could  not  be 
determined.  As  noted  earlier,  it  is  assumed  the  samples  are 
representative  of  the  whole  and  any  prejudice  of  selection 
was  avoided.  The  sample  data  items  had  a  total  estimated 
value  of  $16.9  million,  of  which  $5.2  million  was  found  to 
be  unessential  or  uneconomical ly  acquired  (3:16-19).  If  the 
sample  is  representative  of  the  whole,  then  31%  of  the  data 
items  ($29.3  million)  on  these  26  contracts  are  unessential 
or  uneconomical ly  acquired,  representing  $29.3  million  in 
unnecessary  costs  to  the  government.  However,  since  a 
different  number  of  data  items  from  each  contract  were 
examined,  the  sample  data  needs  to  be  weighted  before  such  a 
conclusion  can  be  drawn. 

The  number  of  sample  data  items  taken  from  each 
contract  was  unrelated  to  contract  value  or  the  total  number 
of  data  items  on  the  contract.  Therefore,  it  is  necessary 
to  weight  the  sample  data  items  so  any  single  contract  or 
data  item  is  not  allowed  undue  influence.  To  accomplish  the 
weighting  of  the  data  items  the  following  procedure  was 
followed.  First,  since  all  of  the  contracts  were  of  a 
different  value  it  was  necessary  to  determine  the  average 
value  of  each  contract.  This  was  done  by  computing  a  simple 
average  for  the  26  contracts  involved.  Since  the  total 
value  of  the  contracts  was  $5.8  billion,  the  average 
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contract  value  was  $233  million.  Next,  an  average  data 
value  for  each  contract  was  determined.  Since  each  contract 
contained  a  different  number  of  data  items,  it  was  necessary 
to  compute  a  weighted  average.  This  was  done  by  dividing 
the  number  of  data  items  for  a  contract  by  the  total  number 
of  data  items  in  the  26  contracts  and  multiplying  this  by 
the  contract's  data  value.  By  summing  these  values  for  all 
26  contracts,  the  weighted  average  data  costs  are  computed 
to  be  $5.23  million  per  contract.  Next,  the  weighted 
average  for  the  sample  data  items  were  computed.  The  method 
of  computation  is  identical  to  the  weighted  averages 
computed  above.  The  weighted  average  of  data  per  sample 
contract  is  $603,418.  Similarly,  the  weighted  average  of 
the  unessential  sample  data  per  contract  is  $108,861  and  the 
weighted  average  of  the  uneconomical  sample  data  per 
contract  is  $187,240. 

With  the  weighted  average  information,  the  percentage 
of  the  sample  data  that  is  unessential  is 

$127,535/$603,418  =  21% 

and  the  percentage  of  the  sampled  data  that  is 
uneconomica  1 ly  acquired  is 

$187,240/$603,418  *  31% 

Thus.  52%  of  the  data  in  the  sample  was  unessential  or 
uneconomical ly  acquired.  Provided  that  the  sample  was 
representative  of  the  26  contracts  examined,  it  can  be 
determined  that  52%  of  the  cost  of  the  data  items  in  these 
contracts  was  incurred  unnecessarily.  This  amounts  to  $49.5 
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million  of  costs  the  government  could  have  avoided.  In 
other  terms,  the  Air  Force  estimated  that  in  fiscal  year 
1986  over  $2  billion  was  used  to  purchase  data  from 
contractors  (3:1).  If  the  sample  is  representative  of  the 
whole,  then  in  1986  over  $1  billion  was  unnecessarily  spent 
on  data. 

Observations .  Comments ,  &  Conclusions .  As  with  any 
analysis,  this  one  contains  what  could  be  considered 
vulnerable  points  that  cause  the  results  to  come  into 
question.  However,  this  analysis  most  likely  underestimates 
the  extent  of  waste  in  data  acquisition,  thus  the  possible 
savings  of  $1  billion  in  1986  is  conservative.  The 
determination  of  unessential  data  was  somewhat  limited  in 
scope.  Contract  data  items  were  reviewed  for  unnecessary, 
excessive,  or  redundant  data  requirements  (3:6),  however 
this  was  a  subjective  determination  on  the  part  of  the 
auditor.  This  method  of  determining  unessential  data  in  no 
way  evaluated  the  accuracy  of  the  data  received  (especially 
critical  for  technical  data),  or  the  benefit  the  government 
derived  from  it. 

Second,  the  cost  to  the  government  of  preparing  data 
requests,  processing,  validating,  storing,  retrieving, 
distributing,  and  subsequently  destroying  this  data  was  not 
included  in  the  reported  data  costs.  Those  attempting  to 
determine  just  how  much  data  really  costs  the  government 
need  to  consider  these  "ownership"  costs  to  the  government 
in  addition  to  the  price  that  is  reported  on  a  DD  Form  1423. 
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Considerations  such  as  the  two  mentioned  above  lead  to  the 


conclusion  that  the  auditor's  report,  while  revealing  a 
substantial  waste  of  $1  billion,  is  only  addressing  one 
aspect  of  the  high  cost  of  data.  With  other  considerations, 
the  real  price  of  data  exceeds  the  amount  the  report 
indicates . 

The  Tea  1  Survey  Analysis 

The  Teal  survey  is  separated  into  two  primary  areas  of 
analysis,  coinciding  with  the  survey's  two  parts.  Part  One 
addressed  "identification  of  problems"  with  the  acquisition 
of  data,  while  Part  Two  addressed  "suggested  improvements" 
to  the  data  acquisition  process.  While  similar  analytical 
tools  are  used  in  each  part,  the  analysis  of  each  part  is 
independent  of  the  other.  This  analysis  centers  around  the 
responses  of  AFLC  and  ASD,  and  the  search  for  similarities 
and  dissimilarities  in  their  responses  to  survey  questions. 
Differences  in  responses  are  sought  since  inter- 
organizational  differences  could  signal  that  different 
problems  are  faced  by  these  organizations.  Knowing  this 
should  assist  in  determining  what  actions  to  take  to 
alleviate  problems  in  the  data  acquisition  process. 

Part  One :  Problem  Identification. 

Overview .  Part  One  consisted  of  17  problem 
statements  which  persons  surveyed  were  asked  to  respond  to 
using  a  Likert  scale.  In  order  to  quantitatively  analyze 
the  responses,  a  value  was  assigned  to  each  possible 
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response.  The  possible 

responses  are 

listed  below. 

along 

with  the  numerical  value 

assigned  to 

each . 

STRONGLY  AGREE 

NEUTRAL 

DISAGREE 

STRONGLY 

AGREE 

5  4 

3 

2 

DISAGREE 

1 

The  responses  to  each  problem  were  first  examined  by 
organization  to  determine  if  the  median  response  to  each 
question  was  significantly  different  from  a  neutral 
response.  This  was  done  to  identify  those  problems  that 
AFLC  and  ASD  answered  as  neutral.  Since  a  neutral  response 
reveals  little  or  no  information,  by  isolating  those  problem 
statements  both  organizations  were  neutral  on,  this  research 
could  focus  on  those  non-neutral  problem  statements.  Note 
however,  that  if  a  particular  problem  statement  elicited  a 
neutral  response  from  only  one  organization,  then  the 
neutral  response  is  worth  investigating  to  determine  why  one 
organization  was  neutral  and  another  was  not  neutral.  After 
determining  neutrality  of  responses,  this  analysis 
concentrated  on  finding  those  statements  where  responses 
significantly  differed  from  organization  to  organization. 
Also  of  interest,  was  the  ranking  of  the  problem  statements 
from  the  strongest  agreement  to  the  strongest  disagreement 
to  determine  which  problems  AFLC  and  ASD  felt  the  strongest 
about.  By  doing  this  it  was  possible  to  rank  the  problem 
statements  in  the  order  of  perceived  severity.  A 
reliability  test  was  performed  on  the  problem  statement 
responses  and  the  suggested  improvements  responses.  The 
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reliability  of  the  problem  statement  responses  was 
Cronbach’s  alpha  =  0.8792,  and  the  reliability  of  the 
improvement  responses  was  Cronbach's  alpha  =  0.8908.  These 
reliability  coefficients  are  high  enough  to  indicate  a 
likelihood  similar  responses  would  be  obtained  if  these 
questions  were  asked  in  a  repeat  of  this  survey.  (The 
reliabilities  were  computed  using  SPSS-X  and  the  procedure 
is  described  in  Chapter  Two.  The  problem  and  improvement 
analysis  used  and  the  results  are  related  in  the  remainder 
of  Chapter  Four.) 

Deviations  From  Neutra 1 .  To  determine  if  a 
problem  statement  was  significantly  different  from  the 
neutral  response,  the  Wilcoxon  signed-rank  test  (18:344-349) 
was  used  to  test  the  response  median  and  compare  it  to 
median  =  3  (or  Neutral).  The  Wilcoxon  signed-rank  test  was 
performed  on  STATISTIX  (25:6.7)  for  each  of  the  17  problem 
statements.  The  results  are  summarized  in  Table  2  below. 

To  be  80%  confident  the  responses  were  different  from 
neutral.  P-values  <  0.20  indicate  the  responses 
significantly  differed  from  neutral.  Since  only  Statement  6 
was  answered  neutral  by  AFLC  and  ASD,  it  will  not  be  given 
consideration  in  further  analysis  and  is  only  included  in 
subsequent  tables  for  informational  purposes.  Problem 
Statement  6  states.  "Government/contractor  efforts  expended 
on  data  pricing  exceed  the  value  added  benefits.” 
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TABLE  2 


Teal  Survey  Problem  Statement 
Significantly  Pifferent  From 
Using  Wilcoxon  Signed-Rank 

Responses 
"Neutral " 
Test 

AFLC 

ASP 

Problem 

Pifferent 

Pifferent 

Statement 

from 

from 

Number 

P-va 1 ue 

Neutra  1 

P-va 1 ue 

Neutral 

1 

.  1747 

yes 

.8025 

no 

2 

.  0003 

yes 

.0004 

yes 

3 

.  1178 

yes 

.  1453 

yes 

4 

.0001 

yes 

.0000 

yes 

5 

.0000 

yes 

.0000 

yes 

6 

.5422 

no 

.7413 

no 

7 

.0000 

yes 

.0001 

yes 

8 

.0000 

yes 

.0001 

yes 

9 

.0073 

yes 

.  1812 

yes 

10 

.0000 

yes 

.0196 

yes 

11 

.0350 

yes 

.  1482 

yes* 

12 

.0026 

yes 

1.0000 

no 

13 

.0001 

yes 

.0030 

yes 

14 

.0003 

yes 

.8789 

no 

15 

.0006 

yes 

.0917 

yes 

16 

.  0000 

yes 

.0004 

yes 

17 

.  0002 

yes 

.0132 

yes 

yes  *  a  significant  difference  from  the  neutral  response. 

Unless  otherwise  noted,  all  yes  responses  agreed  the 
statements  represented  problems. 

no  *  the  responses  were  not  significantly  different  from 
neutra 1 . 

*  Response  significantly  below  neutral,  i.e.  disagreed 
that  this  statement  represented  a  problem. 


AFLC  and  ASP  Pi f f erences .  Having  eliminated  those 
problem  statements  not  significantly  different  from  neutral, 
the  analysis  turned  toward  finding  those  statements  where 


58 


AFLC  and  ASD  significantly  differed  in  their  responses. 

Note,  this  is  not  necessarily  limited  to  dichotomous 
responses  (where  one  agreed  and  the  other  disagreed) .  but 
includes  significant  differences  in  extent  of  agreement  or 
disagreement.  To  determine  if  the  AFLC  and  ASD  responses  to 
each  statement  were  different,  the  nonparametric  Wilcoxon 
rank-sum  test  was  used  (25:675-679).  The  rank-sum  test  was 
performed  using  5PSS-X  (26:809-34).  To  attain  an  80% 

TABLE  3 

Test  of  Differences 

in  AFLC  &  ASD  Problem  Statement  Responses 


Problem  Significantly 

Statement  No.  P-va 1 ue  Pi f f erent 


1 

.  6034 

no 

2 

.7181 

no 

3 

.6442 

no 

4 

.3509 

no 

5 

.6667 

no 

6 

.8288 

no 

7 

.0312 

YES 

8 

.  1236 

YES 

9 

.  2324 

no 

10 

.0227 

YES 

11 

.  0084 

YES 

12 

.0112 

YES 

13 

.  2566 

no 

14 

.0074 

YES 

15 

.3623 

no 

16 

.4991 

no 

17 

.4796 

no 

P-value  <  0.20  indicates  significantly  different  responses 
using  Wilcoxon  Rank-Sum  Test. 
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significance  level  for  the  responses,  any  P-values  <  0.20 
indicates  a  difference  in  the  responses.  In  other  words, 
there  is  an  80%  confidence  that  the  responses  are  different 
Table  3  shows  the  P-values  generated  as  a  result  of  testing 
for  equal  medians. 

AFLC  and  ASD  had  significantly  different  responses  to 
six  of  the  problem  statements.  With  the  exception  of 
Problem  Statement  11.  AFLC  and  ASD  agreed  that  the  survey 
statements  identified  problems,  but  differed  significantly 
in  their  perceptions  of  problem  severity.  On  Statement  11, 
ASD  disagreed  that  this  was  even  a  problem,  while  AFLC 
agreed  this  was  a  problem.  The  six  problem  statements  are 
listed  below  and  continue  on  the  following  page. 

Problem  Statement  7.  There  is  a  lack  of  guidance  and 
training  in  determining,  defining,  and  tailoring  data 
requirements . 

Problem  Statement  8.  There  is  a  lack  of  guidance  and 
training  in  determining,  defining,  and  tailoring  of 
regulations,  specifications,  standards,  etc. 

Problem  Statement  10.  There  are  few  effective 
incentives  for  contractors  to  recommend  data 
elimination,  tailoring,  or  alternatives  to  meet 
government  requirements. 

Problem  Statement  11.  The  data  acquisition  process  is 
inefficient,  inconsistent,  and/or  poorly  executed  and 
contributes  to  poor  quality  RFPs . 
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Problem  Statement  12.  Poor  communications  exist 


between  the  Program  Manager,  team  members,  supporting 
organizations  (both  internal  and  external),  and 
contractors,  which  contributes  to  identifying  data 
requirements  in  an  untimely  manner. 

Problem  Statement  14.  Existing  policies  and  procedures 
do  not  incentivize  SPOs  and  staffs  to  streamline  data 
requirements . 

For  the  six  questions  listed  above,  the  central 
tendency  of  AFLC ' s  answers  showed  a  significant  difference 
(in  all  cases  AFLC  exceeding  ASD)  from  ASD  responses. 
Statements  12  and  14  found  AFLC  suggesting  that  these  were 
problems,  while  ASD  offered  a  neutral  response. 

The  implications  of  the  differences  between  AFLC  and 
ASD  on  problem  statements  7,  8,  and  10  may  not  be 
immediately  obvious  since,  despite  a  significant  difference 
in  responses,  both  organizations  agree  these  are  problems. 
Examination  of  the  content  of  statements  7  and  8,  shows 
these  problems  deal  with  preparations  to  qualify  people  to 
work  in  the  data  management  arena.  It  can  be  inferred  from 
the  responses  that  AFLC  feels  that  training  of  persons  in 
the  data  acquisition  process  is  more  urgent  than  does  ASD. 
This  difference  of  opinion  as  to  the  severity  of  the  problem 
could  influence  the  amount  of  formal  training  a  command  or 
division  is  willing  to  provide  its  people. 

Problem  statement  10  deals  with  incentives  (or  lack 
thereof)  for  contractors  to  take  the  initiative  in 
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streamlining  the  data  acquisition  process.  AFLC  views  this 
as  significantly  more  serious  than  does  ASD .  however  both 
agree  it  is  a  problem.  The  implications  of  this  could  be 
quite  troublesome  if  the  goverr nent  is  to  capitalize  on  the 
benefits  such  incentives  could  bring.  Since  ASD  has  the 
primary  responsibility  in  the  RFP  and  SOW  preparation,  it 
will  likely  take  a  concentrated  effort  on  their  part  to 
incorporate  contractor  incentives  into  RFPs  and  SOWs .  If 
ASD  does  not  view  this  as  a  severe  problem,  then  modifying 
the  process  to  include  contractor  incentives  is  unlikely. 
AFLC  is  not  in  a  position  in  the  acquisition  arena  to 
exercise  controlling  authority  over  the  RFP  and  SOW,  and 
therefore  will  have  to  wait  for  ASD  (or  AFSC)  to  take  the 
initiative . 

Problem  statement  11  is  the  only  statement  where  AFLC 
and  ASD  disagreed  as  to  whether  or  not  it  was  a  problem. 

ASD  did  not  believe  this  to  be  a  problem,  while  AFLC  said  1 
was  a  problem.  The  problem  statement  reads,  "The  data 
acquisition  process  is  inefficient,  inconsistent,  and/or 
poorly  executed  and  contributes  to  poor  quality  RFPs." 

Since  ASD  respondees  are  more  responsible  for  these  aspects 
of  the  process  than  are  AFLC,  there  is  likely  a  reluctance 
on  the  part  of  ASD  respondees  to  acknowledge  any  shortfalls 
on  their  parts.  Assuming  there  was  no  prejudice  such  as 
this,  this  is  an  area  where  further  investigation  should 
take  place.  The  ASD  response  implies  that  the  data 
acquisition  process  is  efficient,  consistent,  properly 
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executed,  and  contributes  to  high  quality  RFPs .  However  the 
ASD  responses  to  statement  5  were  in  overwhelming  agreement. 
Statement  5  reads.  "Program  requirements  which  are  poorly 
defined,  unstable,  inconsistent,  incomplete,  or  over 
specified  during  the  RFP  preparation  phase  can  lead  to 
inappropriate  data  requirements."  ASD  responses  to 
statements  5  and  11  are  contradictory .  This  type  of 
contradiction  illustrates  the  difficulty  there  is  in 
focusing  in  on  actions  to  take  in  future  process  reforms. 

Problem  statement  12  reads,  "Poor  communications  exist 
between  the  Program  Manager,  team  members,  supporting 
organizations  (both  internal  and  external),  and  contractors, 
which  contributes  to  identifying  data  requirements  in  an 
untimely  manner."  This  statement  evoked  a  neutral  response 
fr im  ASD  and  a  definite  agreement  from  AFLC .  This  could  be 
du?  to  the  separate  roles  the  two  organizations  play  in  the 
acquisition  process.  AFLC  is  a  supporting  organization  and 
subsequently  differs  in  its  perspective.  The  Auditor 
General's  Report  cited  earlier  in  this  thesis  found  that  a 
lack  of  adequate  response  time  for  supporting  organizations 
contributed  to  an  inefficient  process  (3:8).  ASD  and 
those  organizations  that  take  the  lead  in  RFP  and  SOW 
preparations  need  to  be  aware  that  the  other  players  in  the 
process  recognize  this  communication  problem.  ASD  needs  to 
recognize  it  also. 

Problem  Statement  14  reads,  "Existing  policies  and 
procedures  do  not  incentivize  SPOs  and  staffs  to  streamline 
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data  requirements."  Since  ASD  responded  with  a  neutral  to 
this  statement,  it  is  difficult  to  determine  just  what  their 
position  is,  and  to  make  too  many  assumptions  could  be 
misleading.  However,  their  lack  of  disagreement  assures 
their  position  that  they  do  not  believe  incentives  currently 
exist.  It  could  also  signal  a  belief  on  the  part  of  ASD 
respondees  that  incentives  for  SPOs  and  staffs  would  be  of 
little  value.  AFLC  on  the  other  hand  agreed  that  current 
policies  do  not  incentivize  streamlining.  It  cannot  be 
assumed  however  that  AFLC  believes  that  streamlining 
incentives  should  be  implemented  for  SPOs.  From  the 
responses,  this  is  an  area  where  further  investigation 
should  take  place  before  any  incentive  policies  are 
instituted . 

The  Most  Severe  Problems .  In  the  following 
section  the  problems  are  ranked  according  to  their  median 
values.  (See  Table  4.)  This  shows  the  relative  importance 
or  severity  AFLC  and  ASD  placed  on  each  problem.  From  fable 
4.  it  appears  that  AFLC  and  ASD  relative  ranking  of  the 
problems  is  similar.  AFLC  and  ASD  have  ranked  four  of  the 
same  problems  in  the  top  five,  and  eight  in  the  top  nine. 
These  problems  are  statement  numbers  4,  5,  7,  8.  10.  13,  16. 
and  17.  By  examining  the  content  of  these  problem 
statements,  it  was  possible  to  group  the  problems  into 
subject  areas.  Statements  4,  5,  and  17  all  address  the 
issue  of  the  government  ordering  too  much  data.  The  reasons 
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TABLE  4 


Ranking  of  Problems  by  Survey  Problem  No.  from 
Most  to  Least  Severe  by  Organization 
(Ranked  by  Median) 


ASD 


AFLC 


5 

5 

4 

7+ 

8  + 

8  + 

7  + 

10  + 

16 

4 

17 

16 

n 

4- 

12  + 

13 

17 

10*  + 

13 

3 

9 

9 

14+ 

15 

15 

6* 

2 

14*  + 

3 

12*  + 

11  + 

1* 

1 

11+  ** 

6* 

+  AFLC  and  ASD  gave  significantly  different  responses. 
*  Neutral  response. 

**  Not  found  to  be  a  problem. 


for  this  include  poor  quality  of  the  RFP  (statement  5).  the 
lack  of  attention  to  data  management  issues  (statement  17), 
and  a  proliferation  of  data  requirements  leading  to 
increased  costs  and  wasted  contractor  as  well  as  government 
efforts  (statement  4) . 

Problem  statements  7,  8,  and  13  address  training 
issues.  The  lack  of  training  and  guidance  addressed  in 
these  statements  were  cited  as  some  of  the  top  problems 
encountered  in  acquiring  data.  While  statements  7  and  8 
were  found  to  be  answered  significantly  different  by  AFLC 


and  ASD  in  the  previous  section  of  this  thesis,  they  are 
both  ranked  in  the  top  four  problems.  This  evidence  that 
both  organizations  identified  these  as  top  problems  should 
diminish  the  significance  of  the  central  tendencies  of  their 
answers  to  be  different.  This  means  that  despite  the 
differences  found  between  Statements  7  and  8,  training  is  a 
problem  that  according  to  AFLC  and  ASD  deserves  immediate 
attention . 

Problem  statements  10  and  16  cite  the  lack  of  incentives 
for  contractors  to  take  the  initiative  in  eliminating  or 
reducing  data  as  a  major  problem.  The  implication  here  is 
that  the  current  situation  encourages  the  contractor  to  not 
reduce  the  amount  of  data  the  government  asks  for  since  the 
contractor  is  getting  paid  for  all  deliveries.  Another 
implication  is  that  the  contractor  knows  what  and  where 
reductions  could  and  should  be  made,  but  as  long  as  he  is 
being  paid  to  deliver,  there  is  no  reason  for  him  to  reduce 
his  income.  Since  the  government  is  so  poor  at  determining 
appropriate  data  costs  (as  was  evident  from  the  Auditor's 
Report),  the  contractor  can  use  this  "hidden  income"  as  a 
management  reserve.  This  would  be  especially  true  of  fixed 
price  efforts,  where  the  high  risk  is  on  the  contractor  to 
perform  the  contract  within  the  dollars  provided. 

Factor  Analysis  of  Problem  Statements .  So  far. 
the  analysis  used  has  resulted  in  a  hypothesis  that  the  most 
severe  problems  in  the  data  acquisition  area  are  described 
by  problem  statements  4.  5,  and  17  (excessive  data  requested 


66 


by  the  government);  7,  8,  and  13  (a  lack  of  training);  and 
10  and  16  (no  incentives  to  minimize  data  requests.)  The 
logical  groupings  were  made  from  intuitively  combining 
statement  rank  and  content.  To  confirm  that  there  are 
correlations  such  as  the  ones  suggested  above,  a  factor 
analysis  was  performed  on  the  responses  to  the  problem 
statements.  SPSS-X  was  used  to  perform  the  factor  analysis. 
When  factor  analysis  is  applied  to  the  responses  of  all  of 
the  problem  statements,  four  factors  were  found.  The 
groupings  found  were: 

Factor  One:  1,  2,  3.  6,  9 
Factor  Two:  7,  8,  12.  13,  17 
Factor  Three:  4.  5.  15,  16 
Factor  Four:  10,  11.  14 

When  factor  analysis  is  applied  to  only  the  top  nine 
problems,  as  determined  by  the  median  ranks,  only  two 
factors  were  found.  The  factor  groupings  are: 

Factor  One:  2,  4,  7,  8,  13,  17 
Factor  Two:  5,  10.  16 

Since  problem  statements  7.  8,  13,  and  17  are  grouped 
in  the  same  factor  for  both  iterations,  it  appears  there  is 
a  correlation  in  the  responses  to  these  problems.  The 
groupings  selected  in  the  previous  section  were  a  result  of 
examining  statement  content  and  attempting  to  group  the 
questions  according  to  subject  area.  Since  statements  4  and 
5.  and  statements  10  and  16  are  in  the  same  factor  at  least 
once,  this  indicates  an  assurance  that  the  groupings 
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previously  selected  are  internally  correlated.  Problem 
statement  17  was  originally  grouped  with  excessive  data,  but 
the  factor  analysis  shows  a  stronger  correlation  with  the 
training  group.  While  problem  17  deals  with  obsolete 
requirements  and  unnecessary  data,  the  respondees  could  have 
attributed  these  circumstances  to  a  lack  of  training.  The 
conclusion  is  that  the  factor  analysis  showed  the  logical 
groupings  of  the  problem  statements  from  the  previous 
section  was  reasonable. 

Summary .  The  Teal  survey  addressed  many  problems 
relating  to  data  acquisition.  While  some  of  these  problems 
elicited  neutral  responses  from  survey  takers,  and  some  were 
found  to  have  significantly  different  answers  from  AFLC  to 
ASD,  three  problem  areas  clearly  emerged  from  the  survey 
results.  One,  the  government  orders  too  much  data  from  the 
contractor,  and  the  survey  found  reasons  for  this  are  poor 
quality  RFPs  and  a  lack  of  attention  given  to  such  data 
management  issues.  Second,  there  is  a  lack  of  training  of 
government  personnel  to  properly  prepare  RFPs  and  SOWs . 
which  naturally  leads  to  the  first  problem.  Third,  the 
government  has  not  given  the  contractor  the  incentive  to 
reduce  the  amount  of  data.  For  the  government  to  address 
the  ever  increasing  cost  of  data,  these  problems  will  have 
to  be  examined  and  solutions  to  them  must  be  found. 
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PART  TWO:  IMPROVEMENTS . 


OVERVIEW .  Analysis  of  the  Teal  survey  continues 
with  the  second  half  of  the  survey  which  asked  the 

* 

respondees  to  give  their  opinion  as  to  the  possible  benefits 

*  of  suggested  improvements  to  the  data  acquisition  process. 
The  list  of  suggested  improvements  is  in  Appendix  C.  Those 
surveyed  were  asked  to  respond  to  a  Likert  scale  for  their 
answers.  This  research  assigned  a  numerical  value  to  each 
point  on  the  scale  to  facilitate  analysis  of  the  responses. 
The  Likert  Scale  and  values  assigned  are  listed  below. 

VERY  HIGH  HIGH  MODERATE  LOW  NONE 

5  4  3  2  1 

A  response  of  VERY  HIGH  means  the  respondee  believes  the 
suggested  improvement  has  the  potential  for  significantly 
improving  the  current  data  acquisition  process,  while  a 
response  of  none  means  no  improvement  to  the  process  can  be 
achieved  through  this  suggestion.  The  responses  to  this 
portion  of  the  survey  will  be  analyzed  using  the  Wilcoxon 
rank-sum  test  to  determine  any  difference  of  central 
tendency  of  the  AFLC  and  ASD  responses.  This  will  reveal 
those  suggested  improvements  where  AFLC  and  ASD 
signi f icant ly  disagree  as  to  the  benefit  of  a  suggestion. 

» 

It  can  be  assumed  that  those  suggestions  where  the  responses 

•  are  not  significantly  different  find  AFLC  and  ASD  of  like 
mind.  Where  differences  are  found,  the  suggestion  is 
repeated  and  implications  of  the  disagreement  are  made.  The 
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30  suggested  improvements  are  also  ranked  by  medians 
reflecting  the  relative  value  each  had  according  to  AFLC  and 
ASD .  After  identifying  those  improvements  where  AFLC  and 
ASD  have  differences  of  opinion  as  to  possible  benefit,  and 
examining  the  ranking  of  the  suggestions  by  AFLC  and  ASD, 
those  improvements  found  to  have  the  most  support  will  be 
discussed.  A  factor  analysis  is  next  performed  on  those 
problem  statements  and  suggested  improvement  responses  that 
were  shown  to  be  of  the  most  concern  to  the  respondees. 

This  will  be  done  to  determine  if  the  improvements  that  were 
most  strongly  suggested  correlate  with  the  worst  problems. 

AFLC  and  ASD  Pi f  f erences .  The  Wilcoxon  rank-sum 
was  used  to  determine  if  the  central  tendency  of  the  AFLC 
responses  differed  from  the  central  tendency  of  the  ASD 
responses.  SPSS-X  was  used  to  perform  the  test  and  the 
results  are  listed  below  in  Table  5.  The  reader  should 
understand  that  a  significantly  different  response  does  not 
necessarily  mean  AFLC  and  ASD  hold  dichotomous  views,  but 
rather  a  difference  here  refers  to  degree  of  the  benefit 
each  thought  possible.  This  information  is  valuable  since 
identification  of  these  differences  gives  those  working  m 
this  area  an  idea  of  what  differences  need  resolution. 

The  improvements  AFLC  and  ASD  differed  significantly  on 
were  suggestions  3,  11,  12.  14,  17.  19,  20,  21.  22.  27.  and 
28.  These  are  examined  following  Table  5. 
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TABLE  5 


Test  for  Differences 
in  AFLC  &.  ASD  Responses  to  Improvements 


Suggested 

Significantly 

Improvement 

P-va lue 

Di f  f erent 

1 

.3102 

no 

2 

.2509 

no 

3 

.0617 

YES 

(ASD) 

4 

.3791 

no 

5 

.  8662 

no 

6 

.  2486 

no 

7 

.  9461 

no 

8 

.  2188 

no 

9 

.  9345 

no 

10 

.6686 

no 

11 

.  1924 

YES 

(AFLC) 

12 

.0367 

YES 

( AFLC ) 

13 

.3894 

no 

14 

.  0349 

YES 

(AFLC) 

15 

.5379 

no 

16 

.9805 

no 

17 

.  1307 

YES 

(AFLC) 

18 

.6285 

no 

19 

.  0002 

YES 

(A^LC) 

20 

.0031 

YES 

(AFLC) 

21 

.0361 

YES 

( AFLC ) 

22 

.0001 

YES 

(AFLC) 

23 

.  8967 

no 

24 

.  2261 

no 

25 

.  5938 

no 

26 

.  6496 

no 

27 

.  0978 

YES 

(AFLC) 

28 

.  0552 

YES 

( AFLC) 

29 

.4544 

no 

30 

.  5371 

no 

P-va lue  <  0.20 

indicates  significantly 

di f  f erent 

(  )  Designates 

organization  that  felt 

strongest  about  the 

suggested 

improvement  being  benef 

icia  1  . 
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Improvement  3  (IM3):  "Emphasize  digital  delivery  of 
cost  and  pricing  data,  proposal  data,  and  other  deliverable 
data."  This  improvement  drew  a  high  rating  from  ASD  but 
only  moderate  support  from  AFLC .  Perhaps  AFLC  and  ASD  use 
deliverable  data  in  different  ways,  where  digital  delivery 
is  not  quite  so  attractive  to  AFLC.  Since  a  possible  change 
to  the  data  acquisition  process  is  in  the  form  oi 
deliverable  data,  this  area  should  be  examined  further  to 
ensure  AFLC  will  be  able  to  use  the  data  in  digital  form. 

IM11:  "Schedule  has  been  a  major  reason  management  has 

been  willing  to  accept  poorly  defined  data  requirements. 
Management  must  realistically  schedule  enough  time  for  data 
requirements  definition."  This  improvement  drew  strongest 
support  from  AFLC,  and  it  echoes  the  subject  addressed  in 
problem  statement  12  where  AFLC  and  ASD  offered  similar 
answers.  Again,  it  appears  correction  of  this  problem  and 
implementation  of  this  improvement  are  the  primary 
responsibility  of  ASD  (AFSC). 

IM1 2 :  "Establish  a  cost  accounting  standard  (CAS) 

guideline  for  pricing  data."  AFLC  thought  this  suggestion 
would  offer  a  high  payoff,  while  ASD  responded  with  a  low 
moderate  rating.  This  suggestion  would  add  standardization 
to  the  data  pricing,  although  ASD  is  questioning  the  payoff 
of  this  action. 

IM1 4 :  "Develop  mandatory  training  courses  for  data 

managers  and  RFP  writers."  A  high  rating  was  given  by  AFLC 
and  a  moderate  to  high  rating  by  ASD.  This  drew  generally 
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large  support  from  both  groups,  although  AFLC  anticipates 
higher  payoffs.  This  need  for  additional  training  agrees 
with  the  findings  of  the  studies  looked  at  in  Chapter  Three 
and  the  identification  of  lack  of  training  as  a  severe 
problem  in  the  first  half  of  this  survey.  A  future  study  to 
find  out  how  much  formal  training  current  data  managers  and 
RFP  writers  have  would  be  helpful  in  determining  where  and 
what  training  to  prepare  for  future  managers. 

IM17:  "Ensure  wording  is  included  in  SOW  to  implement 

the  acquisition  streamlining  clause."  ASD  gave  this  only  a 
low  to  moderate  chance  of  improving  the  process,  while  AFLC 
viewed  the  chances  as  being  moderate  to  high.  This  is  a 
curious  response  since  AFLC  agreed  in  Problem  statement  10 
that  the  contractor  had  few  incentives  to  streamline. 

Without  incentives  to  streamline,  it  is  doubtful  a 
contractor  will  take  this  action  regardless  of  wording  in 
the  SOW.  ASD ' s  low  rating  could  be  a  result  of  past 
streamlining  rhetoric  that  has  yet  to  have  a  measurable 
positive  impact  on  the  process. 

IM1 9 :  "Separately  price  each  data  item."  ASD  gave 

this  suggestion  a  low  rating,  while  AFLC  came  in  with  a  high 
racing  or.  this  item.  The  difference  in  perceived  value  of 
separately  pricing  each  data  item  should  be  examined 
further,  especially  since  both  organizations  were  neutral  on 
the  value  added  benefits  of  data  pricing.  (Recall  Problem 
Statement  6.)  It  should  be  determined  why  there  is  such  a 
sharp  contrast  in  responses.  Perhaps  ASD  has  some 
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experiences  not  shared  by  AFLC .  Data  pricing  is  difficult 
though  recent  studies  (such  as  the  ones  in  Chapter  Tnree ) 
and  suggested  actions  from  the  Armed  Forces  Pricing  Manual 
(10:9-29)  indicate  a  move  toward  this  practice.  The 
differences  that  ASD  and  AFLC  have  over  this  issue  should  be 
explored  further,  before  regulations  are  changed  or  enforced 
to  demand  separate  pricing.  The  data  pricing  issue  should 
be  investigated  further  to  determine  if  there  are  categories 
of  data  vtnere  pricing  could  yield  more  benefits  than  others. 

IM20 :  "Increase  approval  level  for  exceptions  to 

separately  pricing  data  for  added  visibility."  Since  ASD 
did  not  find  the  idea  of  separately  priced  data  appealing, 
they  found  this  suggestion  even  less  worthwhile.  With  a 
rating  of  none  to  low,  ASD  found  little  or  no  value  in 
elevating  the  level  at  which  the  decision  to  not  separately 
price  each  data  item  would  be  made.  AFLC  continued  moderate 
support  for  involving  a  higher  level  of  approval  authority. 

IM2 1 :  "Devise  automated  centralized  systems  for 
recording/analyzing  data  prices."  ASD  increased  its 
perception  of  the  usefulness  of  data  pricing  when  the 
prospect  of  an  automated  system  for  pricing  was  made 
available.  Perhaps  the  lack  of  enthusiasm  ASD  expressed  for 
data  pricing  in  IM19  was  driven  by  the  prospect  of  manually 
determining  the  price  of  data.  An  automated  system  was 
given  a  moderate  rating  by  AFLC.  The  whole  issue  of  pricing 
data  did  not  meet  with  a  lot  of  enthusiasm  from  AFLC  or  ASD, 
though  AFLC  thought  it  carried  more  promise  than  did  ASD. 


7  A 


IM22:  "Establish  a  standard  data  package/draft  CDRL 

that  is  common  to  most  or  all  programs  within  a  product 
division."  ASD  gave  this  suggestion  no  better  than  a  low 
chance  of  improving  the  data  acquisition  process  while,  AFLC 
rated  this  suggestion  a  very  solid  HIGH.  This  action  was 
taken  by  ESD  and  was  favorably  received  (12). 

IM27 :  "Establish  a  data  management  training  program 

with  industry."  While  AFLC  answered  significantly  higher 
than  did  ASD.  they  still  only  judged  this  improvement  to 
have  moderate  payoff.  In  looking  at  the  responses  to  IM14, 
where  training  was  held  in  high  regard  by  both  AFLC  and  ASD, 
when  the  prospect  of  training  with  industry  was  approached, 
both  showed  skepticism  as  to  the  likelihood  of  offering 
improvement . 

IM28:  "Recommend  AFSC  develop  a  computer-aided  guide 

on  how  to  tailor  data."  AFLC  again  felt  a  high  payoff  was 
available,  while  ASD  offered  no  better  than  a  moderate 
prospect.  This  question  was  worded  in  such  a  way  as  to  not 
know  whether  respondees  were  responding  to  the  usefulness  of 
a  computer-aided  guide,  or  the  appropriateness  of  AFSC 
developing  such  a  guide. 

The  Most  Benef ic ia 1  Improvements .  To  put  the 
differences  in  AFLC ' s  and  ASD 1 s  answers  into  yet  another 
perspective,  the  suggested  improvements  are  ranked,  by 
organization,  from  the  most  likely  to  be  beneficial  to  the 
least  likely.  The  improvement  rankings  are  listed  in 
Table  6.  The  rankings  were  determined  by  comparing  the 
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medians  of  each  question's  responses.  The  top  ten  responses 
from  AFLC  and  ASD  are  compared  for  common  improvements . 

(Due  to  a  tie  for  AFLC  responses  to  suggested  improvements 
5,  12,  and  30,  the  top  twelve  responses  are  taken  from 
AFLC.)  These  top  ranked  improvements  are  looked  at  further 
to  see  if  they  can  be  grouped  into  improvement  areas,  if  the 
improvements  address  the  problems  noted  in  part  one  of  this 
survey,  and  if  the  improvements  suggested  correlate  to  the 
findings  of  the  studies  in  Chapter  Three.  The  top  suggested 
improvements  that  both  AFLC  and  ASD  felt  would  improve  the 
data  acquisition  process  are  numbers  4,  5,  10,  11,  14.  29. 
and  30.  (The  Teal  survey  Critical  Action  Team  asked  those 
surveyed  to  rank  the  improvements.  A  simple  frequency  count 
determined  the  rankings  (10).  See  Appendix  D  for  their 
results.  In  contrast,  the  analysis  included  here  attempts 
to  group  the  most  beneficial  improvements  into  common  groups 
and  determine  a  link  between  the  most  severe  problems  and 
the  most  beneficial  improvements.) 

The  improvements  can  be  grouped  into  logical  subject 
areas.  Improvements  5.  14,  and  29  all  call  for  additional 
training  for  those  working  in  the  data  acquisition  area. 
Since  lack  of  training  was  found  to  be  among  the  worst 
problems,  it  was  logical  that  suggestions  for  increasing 
training  would  be  among  AFLC ' s  and  ASD ' s  top  improvements. 
The  suggested  improvements  are  repeated  following  Table  6. 
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TABLE  6 


Rankings  of  Suggested  Improvements  From 
Most  to  Least  Beneficial  by  AFLC  and  ASD 


AFLC 

ASD 

14.  11.  10 

3.  10 

29 

4.  29 

4 

11 

13 

8,  14. 

6.  19,  22 

5.  16 

5.  12,  30 

9 

9.  15 

13,  6 

16.  28 

15,  25 

27 

7 

17,  18 

18,  27 

3.  8.  25 

26,  28 

20 

1,  23 

2.  7.  21 

12,  21 

1,  23 

17 

26 

22 

24 

19 

2.  20.  ; 

For  the  text  of  suggested  improvements,  see  Appendix  C. 


Improvement  5.  Provide  an  RFP  preparation  guidance 
document/computer i zed  system  with  lessons  learned  to 
help  in  preparation  of  the  RFP. 
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Improvement  14.  Develop  mandatory  training  courses  for 
data  managers  and  RFP  writers. 

Improvement  29.  Provide  additional  training  to  those 
involved  in  specifying,  reviewing,  inspecting,  and 
accepting  engineering  data. 

Suggested  improvements  10,  11.  and  30  deal  with 
determining  what  data  items  should  be  ordered,  how  they  should 
be  tailored,  and  ensuring  adequate  time  is  allowed  to  do  this. 
All  of  these  have  an  impact  on  the  quantity  of  data  the 
government  orders.  By  implementing  all  three  of  these 
suggestions,  the  quantity  of  data  requested  would  likely 
decrease,  the  quality  increase,  and  subsequently  the  costs  to 
the  government  would  decrease.  It  appears  that  these  three 
improvements  would  alleviate  the  severity  of  the  problems 
identified  by  problem  statements  4,  5,  and  17.  These 
improvements  are  listed  below. 

Improvement  10.  Validate  the  need  for  requirements  (e.g. 
standards,  specs)  derived  from  referenced  documents  and 
require  only  those  that  are  justifiable.  Tailor  to 
program  needs . 

Improvement  11.  Schedule  has  been  a  major  reason 
management  has  been  willing  to  accept  poorly  defined  data 
requirements.  Management  must  realistically  schedule 
enough  time  for  data  requirements  definition. 

Improvement  30.  Primary  data  user  should  request  data. 
Problem  statements  10  and  16  suggest  the  contractor  may 
hold  the  best  answers  to  reducing  the  high  cost  of  data. 
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One  area  where  the  costs  could  be  reduced  Is  addressed  in 
Improvement  4. 

Improvement  4.  Require  contract  proposals  to  identify 
data  that  would  be  more  economically  provided  in 
commercial  format. 

When  examining  problem  and  improvement  statement 
content,  it  appears  the  respondees  have  linked  the  problems 
to  possible  solutions.  However,  as  the  following  section 
shows,  an  analysis  that  depends  on  ranking  the  problems  and 
improvements  by  the  strength  of  the  responses  does  not 
always  assure  a  correlation  exists  between  the  two.  The 
information  revealed  thus  far  is  .important,  because  it  shows 
where  those  working  in  the  data  management  area  perceive  the 
problems  and  solutions  to  be.  However,  if  action  is  ever  to 
be  taken  to  address  some  of  these  problems,  it  is  important 
that  there  be  a  correlation  between  the  problems  and  the 
proposed  solutions.  If  this  correlation  does  not  exist, 
actions  taken  may  not  really  solve  or  alleviate  the 
problems . 

Factor  Ana lys i s  of  Prob 1 ems  and  Improvements .  To 
determine  if  a  correlation  exists  between  the  most  severe 
problems  and  the  most  recommended  improvements,  the  top 
problem  statement  responses  and  the  top  suggested 
improvement  responses  were  examined  using  factor  analysis. 
The  factor  analysis  found  the  factors  indicated  in  Table  7. 
(see  Appendix  F  for  the  rotated  factor  matrix.) 
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TABLE  7 


Comparing  Methods  for  Determining  Relationships 
Between  Problems  and  Improvements 


Intuitively 
Linking  Most  Severe 
Problems  w/Best 
Solutions 


Factor  Analysis  of 
Most  Severe  Problems 
&  Best  Solutions 


Training 

Problems  Solut ions 


P-7  IM-5 

P-8  IM-14 

P-13  IM-29 

Excessive 

Data 

Problems  Solutions 


P-4  IM-10 

P-5  IM-11 

P-17  IM-30 

Lack  of 
Incentives 

Problems  Solutions 

P-10  IM-4 

P-16 


Factor  One 

Prob 1 ems  So  1 ut ions 


P-4  IM-14 
P-7  IM-29 
P-8 


Factor  Two 

Problems  Solutions 


none 


Factor 

Problems 

P-10 

P-16 

P-17 

Factor 

Problems 

P-5 

Factor 

Problems 


IM-4 

IM-5 

IM-10 

IM-30 

Three 

Solutions 


none 

Four 

So  1 ut ions 

IM-11 

Five 

Solut ions 


P-13  none 


P  — 
IM  — 


indicates 

indicates 


problem  statement, 
suggested  improvement. 


the 


If  the  results 
factor  analysis 


of  the  previous  section 
results,  it  can  be  seen 


are  compared 
that  only 


to 
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Factors  One  and  Four  agree  with  the  intuitive  relationships. 

From  this,  several  conclusions  can  be  drawn. 

a.  The  lack  of  training  has  been  established  to 
be  a  problem,  and  the  suggested  improvements  correlate 
with  the  problems.  Policy  makers  should  feel  confident 
that  investing  in  additional  data  manager  training 
would,  in  the  opinion  of  those  working  in  data 
management,  improve  the  data  acquisition  process. 

b.  Factor  Four  shows  a  correlation  between  the 
problem  of  poorly  defined  data,  problem  statement  5, 
and  the  amount  of  time  allowed  to  define  data 
requirements,  improvement  11.  This  correlation  was 
also  found  in  the  ranking  analysis.  Again,  policy 
makers  should  give  strong  consideration  to  demanding 
more  time  be  allowed  for  requirements  definition.  It 
is  most  likely  that  better  defined  data  requirements 
would  lead  to  reduced  costs. 

c.  The  lack  of  incentives  for  the  contractors  to 
reduce  data  did  not  correlate  with  any  of  the  top 
solutions  using  factor  analysis.  Upon  examining  the 
list  of  30  suggested  improvements,  there  are  no 
improvements  that  directly  suggest  contractor 
incentives,  thus  one  of  the  most  severe  problems  could 
not  be  correlated  with  any  specific  actions.  This  is  an 
oversight  of  the  survey  instrument  rather  than  a 
suggestion  that  incentives  are  not  available  to  address 
this  problem. 
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Summary .  In  this  chapter,  the  problems  associated 


with  data  acquisition  have  been  explored.  The  auditor's 
report  analysis  showed  how  easy  it  is  to  discount  the  cost 
of  data  as  insignificant.  Yet  when  the  contracts  were 
examined  using  more  rigorous  techniques,  the  cost  of  data 
swelled  from  the  report's  figure  of  $5.2  million  to  a  more 
realistic  $1  billion.  As  long  as  the  perception  is  that 
data  costs  are  insignificant,  the  urgency  of  policy  makers 
to  take  action  will  not  be  attained.  Data  is  a  cost  driver 
and  must  be  treated  as  such  by  program  managers .  The 
auditor's  report  also  pointed  out  problems  with  the  data 
acquisition  process.  Data  tailoring,  data  calls,  and  DRRBs 
are  not  effectively  implemented  by  program  managers.  If  the 
program  manager  knew  the  real  cost  of  data,  perhaps  these 
functions  would  be  treated  more  seriously  than  the  auditor's 
report  indicates  is  now  being  done. 

The  Teal  survey  attempted  to  find  the  most  serious 
problems  and  the  improvements  that  would  make  data 
acquisition  a  more  efficient  and  effective  process. 

A  call  for  increased  training  of  those  people  working  in  and 
around  data  management  was  declared  a  high  priority. 

This  coincides  with  the  recommendations  of  the  studies 
reviewed  in  Chapter  Three.  From  the  survey,  it  can  also  be 
concluded  that  the  quality  of  government  RFPs  is  a  major 
reason  for  inappropriate  data  being  ordered.  The  main 
reason  for  this,  in  addition  to  the  lack  of  a  properly 
trained  work  force,  is  program  managers  do  not  ensure  enough 
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time  to  accomplish  a  proper  requirements  definition.  This 
was  demonstrated  to  be  a  common  condition  in  the  auditor's 
report . 

The  Teal  survey  contained  several  weaknesses  that  made 
in-depth  analysis  of  other  than  the  most  accented  problems 
and  improvements  impractical.  The  survey  collected  no 
demographic  data  on  those  answering  the  survey.  It  -would  be 
valuable  to  know  how  iong  each  respondee  has  worked  in  the 
data  management  arena,  their  pay-grade,  their  level  of 
formal  training  in  data  management,  and  their  frequency  of 
exposure  to  RFP  and  SOW  preparation.  Since  these  could  not 
be  determined,  it  is  not  possible  to  determine  if  the 
differences  noted  between  AFLC  and  ASD  are  attributable  to 
the  differences  in  the  organizations  or  the  diiioicnce  in 
the  level  of  data  managers  the  survey  may  have  been 
distributed  to  within  the  organizations.  Since  the  sample 
size  was  not  large  compared  to  the  number  of  people  working 
in  the  data  acquisition  area,  there  is  no  assurance  that 
those  surveyed  were  randomly  chosen,  so  the  possibility  of 
selectively  choosing  who  would  answer  was  possible.  It  is 
for  this  reason  that  the  differences  in  the  responses  of 
AFLC  and  ASD  were  not  given  a  more  rigorous  treatment.  This 
is  also  the  reason  that  without  further  investigation  of  the 
differences  noted,  policies  and  procedures  affecting  these 
areas  should  not  be  instituted. 
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V .  Cone  1 us i ons  and  Recommendations 


The  research  for  this  thesis  revealed  many  problems 
that  plague  the  data  acquisition  process.  While  there  are 
numerous  opinions  as  to  what  the  most  severe  problems  are, 
(many  of  which  are  local  in  nature),  this  thesis  found  those 
problems  that  permeate  throughout  the  acquisition  process. 

By  identifying  those  areas  that  everyone  in  the  acquisition 
arena  finds  to  be  problems,  it  becomes  possible  to  address 
those  problems  with  corrective  actions.  Because  all  of  the 
problems  identified  were  not  universally  experienced 
throughout  the  Air  Force  or  DOD,  only  solutions  to  universal 
problems  should  be  addressed  from  an  Air  Force  or  DOD  level. 
From  the  research,  three  major  problems  exist  within  the 
current  data  acquisition  process.  Each  of  these  problems  is 
complicated  by  the  other,  thus  no  one  solution  is  likely  to 
alleviate  all  of  the  problems.  This  research  showed  that 
those  working  in  the  data  management  area  often  have 
different  ideas  as  to  what  solutions  to  implement,  usually 
depending  on  the  where  they  work.  The  remainder  of  this 
chapter  addresses  the  thesis  hypotheses,  summarizes  major 
findings  and  their  impact,  and  makes  suggestions  for  further 
research . 

Prol i feration  of  Data 

Government  program  managers  do  not  know  how  much  data 
requirements  actually  cost  their  program.  Data  prices  are 
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not  known,  data  requirements  are  poorly  written,  and 
excessive  data  (costing  over  $1  billion  annually  for  the  Air 
Force  alone)  is  ordered  by  the  government.  These  themes 
were  found  throughout  this  research.  In  addition,  program 
managers  do  not  take  the  actions  required  to  improve  the 
quality  of  data  ordered,  and  they  do  not  observe  the 
required  data  acquisition  procedures.  Program  managers  do 
not  ensure  data  calls  are  properly  conducted,  nor  do  they 
give  data  calls  or  DRRBs  the  top  management  attention  needed 
to  be  effective.  Because  data  is  a  cost  driver  for  many 
programs,  and  because  the  quality  of  the  data  received  can 
affect  program  success,  program  managers  need  to  elevate  the 
role  of  the  data  manager  to  a  senior  management  level. 

In  addition  to  elevating  the  role  of  the  data  manager, 
there  are  actions  that  can  be  taken  to  eliminate  some  of  the 
confusion  surrounding  data  acquisition.  Standard  data 
packages  need  to  be  established.  These  are  data  items  that, 
after  tailoring  to  a  specific  contract,  can  be  ordered 
without  justification,  thus  time  can  be  spent  examining  non¬ 
standard  data  requests.  Data  needs  to  be  categorized 
according  to  ability  to  price.  Pricing  some  data  items  may 
be  cost  prohibitive  due  to  the  nature  of  the  item,  and  these 
snould  not  be  priced.  However,  most  data  items  can  and 
should  be  priced  and  program  managers  should  demand  prices 
be  delivered.  Further  research  should  examine  the 
feasibility  of  pricing  data  and  attempt  to  determine  pricing 
categories.  Further  research  on  program  managers  attitude 
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toward  data  management  would  also  be  useful .  If  program 
mangers  realize  data  is  a  cost  driver,  and  that  too  much 
data  is  ordered  by  the  government,  then  the  next  step  is  to 
determine  actions  to  alleviate  these  conditions.  However, 
if  the  program  manager  does  not  believe  data  represents  a 
significant  cost,  then  educating  the  program  manager  as  to 
the  real  costs  must  precede  any  actions. 

Contractors  may  hold  the  key  to  the  problem  of 
excessive  data.  This  research  showed  those  working  in  the 
data  management  arena  believe  the  contractor  could  and  would 
help  the  government  reduce  the  quantity  and  cost  of  data  if 
they  were  given  the  incentive  to  do  so.  Currently  the 
contractor  is  allowed  to  charge  for  all  data,  though  he  does 
not  necessarily  have  to  give  a  per  item  price,  since  the 
government  often  waives  this  requirement.  Since  data  is  an 
area  where  the  contractor  can  be  vague,  even  totally  obscure 
estimated  costs  as  "not  separately  priced,”  it  affords  him 
the  opportunity  to  hedge  against  cost  overruns  and  is  an 
opportunity  to  include  unchallenged  costs.  If  the 
contractor  were  incentivized  by  the  government  in  some  way 
to  make  reduction  of  data  in  the  contractor's  interest,  real 
data  reduction  could  be  realized  almost  immediately. 

Further  research  needs  to  be  done  to  determine  what 
incentive  actions  could  be  taken  to  make  data  reduction  a 
goal  of  the  contractor. 
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Training 

A  lack  of  training  of  government  employees  in  the  art 
of  data  acquisition  echoed  as  a  problem  throughout  this 
research.  It  appears  this  lack  of  training  is 
characteristic  of  all  levels  of  personnel  in  all  data 
related  jobs.  From  program  managers  on  down,  personnel  do 
not  possess  the  skills  required  to  ensure  the  government 
efficiently  and  effectively  acquires  data.  Evidence 
suggests  program  managers  do  not  understand  the  process,  or 
the  significance  of  the  costs,  because  they  have  never 
received  formal  training  in  the  data  acquisition  area.  It 
could  be  this  lack  of  training  of  program  managers  that 
keeps  them  unaware  of  the  significance  of  data  to  their 
program's  success. 

Training  is  universally  lacking  for  those  people 
involved  in  data  acquisition.  Virtually  all  of  the  studies 
reviewed  in  this  thesis,  including  the  Teal  survey,  found 
training  to  be  neglected.  This  lack  of  training  could  be 
attributable  to  many  things,  though  the  most  likely  reason 
is  the  variety  of  people  working  data  and  the  frequency  with 
which  they  do  so.  In  the  Air  Force,  the  majority  of  the 
data  management  function  is  performed  by  personnel  working 
in  this  area  part-time.  There  is  likely  a  reluctance  to 
send  part-time  data  managers  to  training.  This  is  an  area 
where  additional  research  should  be  performed.  It  would  be 
beneficial  to  know  how  much  formal  training  full-time  and 
part-time  data  managers  receive. 
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Another  complication  to  training  data  managers  is  the 
variety  of  training  needed,  both  by  the  managers  themselves 
and  by  those  personnel  with  whom  they  interface  in 
determining  data  requirements.  Data  managers  working  on  RFP 
preparation  must  possess  a  different  set  of  skills  from  the 
data  manager  concerned  exclusively  with  tracking  data  on  a 
program.  The  various  people  involved  in  the  data  call 
process  should  all  be  trained  in  the  importance  of  carefully 
selecting  only  necessary,  but  quality  data.  Even  those 
people  involved  only  occasionally  in  the  data  call  process 
should  receive  enough  training  to  understand  the  process,  to 
learn  how  to  evaluate  the  importance  of  the  data  he/she 
requests,  and  to  learn  how  to  tailor  data  requests  so  only 
essential  data  is  acquired.  Until  data  managers  are 
recognized  as  a  career  field  requiring  extensive  skills,  and 
until  they  are  recognized  for  the  critical  role  they  play  in 
determining  a  program's  success,  it  is  likely  available 
training  will  not  noticeably  increase.  Just  as  the  Air 
Force  trains  personnel  before  making  them  pilots  or 
engineers,  formal  acquisition  training  should  be  made 
mandatory  for  all  working  in  the  data  management  area. 
Further  research  here  should  investigate  the  Navy  data 
manager  certification  program.  The  success  of  this  Navy 
initiative  could  reap  dividends  for  the  Air  Force  also. 

The  training  of  data  managers  and  those  involved  in  any 
aspect  of  the  data  acquisition  process  must  involve  more 
than  just  instituting  fancy  buzzwords.  "Streamlining"  and 
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“TQM"  programs  are  doomed  to  failure  if  they  are  not 
followed  up  with  adequate  training.  The  outcry  for  training 
in  the  data  management  area  indicates  a  neglect  on  the  part 
of  senior  level  management  to  provide  sufficient  training. 
The  attitude  toward  training  in  the  DOD  appears  to  be  more 
concerned  with  the  number  of  people  put  through  training 
rather  than  the  quality  of  that  training.  For  instance,  in 
his  book  The  Defense  Management  Challenge,  J.  Ronald  Fox 
points  out  how  the  length  of  training  at  the  Defense  Systems 
Management  College  (DSMC)  is  determined  not  by  the  time 
required  to  disseminate  all  of  the  information,  but  rather 
by  the  limit  DOD  has  imposed  for  temporary  duty  assignments. 
This  time  constraint  has  forced  an  abbreviated  schedule  of 
courses,  thus  DSMC  has  only  one  course-hour  devoted  to  the 
area  of  data  requirements  (16:227-229).  Until  the  DOD 
recognizes  the  importance  of  quality  training  in  all  areas, 
it  is  doubtful  adequate  training  will  be  available  to  data 
managers . 

With  the  advances  in  technology  having  an  ever 
increasing  impact  on  the  acquisition  world,  the  need  for 
training  becomes  more  urgent.  The  initial  stages  of  CALS 
are  being  implemented  throughout  DOD.  This  is  a 
sophisticated  system  that  will  require  extensive  skills  on 
the  part  of  the  government  personnel  if  the  system  is  to  be 
a  success.  This  will  require  not  only  extensive  training, 
but  those  directly  involved  with  CALS  will  need  to  be  in 
more  senior  positions  than  data  managers  currently  occupy. 
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Thus,  a  designated  data  management  career  field,  with 
opportunities  for  senior  level  positions  are  imperative  if 
CALS  is  ever  expected  to  become  a  way  of  doing  business. 

Problems  and  Solutions 

The  data  acquisition  touches  virtually  every  phase  of 
the  acquisition  cycle.  Many  of  the  people  working  in  data 
management  work  only  one  part  of  the  cycle,  and  therefore 
see  only  a  fraction  of  the  problems  encountered  throughout 
the  entire  acquisition  process.  Each  phase  has  its  own 
peculiar  problems  along  with  problems  shared  by  others. 

With  these  differences  comes  different  ideas  on  how  to  solve 
those  problems.  There  is  often  disagreement  between  those 
working  different  phases  of  the  acquisition  cycle  as  to  what 
action  should  be  taken  to  improve  the  process.  AFLC  and  ASD 
did  not  always  agree  on  just  what  the  problems  were  and  what 
improvements  would  alleviate  the  problems.  Policy  makers 
need  to  be  aware  of  these  differences. 

Summary 

While  the  area  of  data  management  is  a  complicated 
network  requiring  the  participation  of  a  diverse  group  of 
government  and  contractor  disciplines,  there  are  actions 
that  can  be  taken  to  alleviate  some  of  the  problems 
surrounding  data  management.  First,  government  program 
managers  must  be  made  aware  of  the  high  costs  of  data. 
Program  managers  must  get  involved  directly  with  the  data 
acquisition  process.  Data  costs  are  a  significant  portion 
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of  a  program  manager's  budget,  and  his/her  attention  to  data 
requests  is  likely  to  save  dollars  and  improve  the  quality 
of  the  information  a  program  office  receives. 

Training  of  virtually  all  persons  involved  in  the  data 
acquisition  process  is  essential.  Those  individuals  working 
this  area  are  responsible  for  billions  of  Air  Force  dollars 
every  year.  With  the  potential  for  saving  over  half  of 
these  costs,  the  training  efforts  offer  the  prospect  of 
immediate  payback.  As  the  data  management  becomes  more 
sophisticated,  the  training  and  competency  of  data  managers 
must  also  increase  or  the  technical  advances  could  be 
compromised  because  of  inadequacies  of  those  operating  and 
managing  the  system. 

Contractors  must  be  offered  incentives  to  reduce  rising 
data  acquisition  costs.  Only  the  contractor  knows  where  the 
greatest  savings  can  be  realized  in  data  acquisition.  The 
government  needs  to  adopt  more  of  a  cooperative  effort  with 
the  contractors  in  order  to  gain  their  aid  in  reducing  data 
acquisition  costs. 
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Appendix  A :  The  Teal  Survey  Raw  Data 

The  following  are  the  responses  to  the  Teal  survey 
problem  statements  and  suggested  improvements.  The  format 
statement  used  with  SPSS-X  was  (F2.0.47F1 .0) .  The  first  two 
columns  were  organization  identifiers.  The  organizations' 
responses  included  in  the  survey  and  their  identifiers  are: 

contractors  =  11 

AFLC  -  12 

HQ  AFSC  =13 

ASD  =  14 

The  next  17  columns  are  the  responses  to  the  17  problem 
statements,  and  the  remaining  30  columns  are  the  responses 
to  the  suggested  improvements.  Blanks  represent  no 
response . 

1143555545553334554342  3  5  4  3  3  5  111  2  4  34 

11443553444334444222  43  4  4  24  41  44 

113355453422344  4321553141  41  1  44  1  5122  514 

114  454455544454445  4434435354344344  22253  334444 
1135355555553555555334453353531554352121112344445 
1153555553554455555355555254452145551111551535555 
123343535525455324433342443325244432  3  21415142 

125  434355333444434343454444323555553334552532455 
1243355455555444455235554352454  5433333343  43344 
1255355455555553555342  5  445454554434  5554544454 

12532555555525555552  45  5  5  5555  453  5445  44  45 

12333442555545433451  524525545544544232545  3243 

12242444444433433343  3442  4435533  3435543533533 

123545544443223244424  4331  4444  54152555251334235 
1223224322134444333324343233433433333433423332434 
1244344455555444444  4  4  444145  33144  4  1151  11  5 

1243244424233223432  4  4  445444  53141  5 

1233344244444424442  4  444  44455445  454  1  424 

1244344333343333433432434444444343244432231423344 
12  55  54  4224244423  554  11  .  5451  1512  4353 

12234452442422344443244433  5544  4  4  42243  4244 

1222255244442444444332334224542254433332523333254 
1245555555552244455322224232445555444112423224445 
1254455454554444445  22333422342243233232232323233 
1244444455443444454344444444454444444332434234453 
12444442444444443443  45434  44  41  4 

124444433244424344421344445443  3434  3  2341  313344 
12243452544444343441  423321435144212241451  4  1555 
121324534544454454414133421512214243354223  15134 

1244344254443443444  34  54  544 

12244342555  2233333  34  32  44  55  55  455  543545 

1224324122242444232443232233345433332555322223223 
125454555554555444544355  34444435511  323125  5 

1213334  53343434343  4341  3414  2 

12434552444544433444144113  334  33 
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1223224333234444343324343233433433333433433333334 
124532434443  54433  34433344  4454  553433  4443 

1233344244343333443  334433  243213224331241  313243 
1233444455444434343111434212321151121445341111343 
123534445445424545413543322  1  5321151113324  3443 

1244444444444444444222424444443344141433231413444 
12  4143433244  4423  545445  42  4  43 

1233344333333233334 
1234334243243443332 

135545544254444244554554352445  32253  442115313235 
1322145421222232332332443454543144  4432454443344 

134423425522243222353534434344  355334223342222451 
1345555455444454445332432244443143323222115314252 
13233353322223333343  523222  22  5  32  2  3421311241 

144425514245153  3442142442555455  55  34  141341353 

1423444333442443455424222134142523121111221343141 
1445344524441142442113334445551443114211211533355 
145555524445534444441453434355  43  5  5113245413323 
141  3332341322221313134212131214  1  211111212213 

144444544422445444454444445445  44444443234  335144 
1424434444342442322334433333443444343332232434443 
1443445444343444344334443342342333232212232233344 
142225522422244444244555444555555454511525251  255 
1422324322242223222223322234222222222222222222222 
1424244344221232442233333332334345444342212424244 
1445455344454333545244434454553354444323333334344 
145355535544424455443443435  4422334322234133211  4 
1412144254222442224214344334443444222234322444455 
1444454344424444444122234333333333422322112322323 
1424444445243454435514341111114411111111411211332 
14142445444442422245134513  555  555  24  4 

14442333  4  123422221  43422334223234432  32  1  2135 

14234452551334445441543252 

14554552555222424444  4453452443345422514132314453 
141454434452422244443542423444  543113111352211113 
143424424423244244342445124543433544243322  34345 
1453452454522242445222225223222252222222252212242 
1423244344242222232322  322322  2232222222112323222 
1434345433344243244524231244322231214223241431243 
1445455244544444345553354444442354324413341433354 
14444453445534243442554553454413524  23  33413  445 

1445245455232442245345443335  55554434334353  33355 
1422434322442234442124433441422111113111111131114 
1454455443554444454  54443  234  444  31  324 

1433444344433433334333333333333333333333333333333 
1444354  544433332442  3444444533344424423244334335 
1443234333333233334  2534  342  34  4 

14335444434433334433344333  3  2 

14232342223322332322 

14255543441523454553  3544553425144144112  15343345 
144424434445313345322444333243  433443433432333434 
1424234344433333333  44444444  44  4  4444311444344 
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Appendix  B :  Tea  1  Survey  Problem  Statements 

The  following  are  the  17  problem  statements  contained 
in  the  Teal  Survey  (28) .  Those  surveyed  were  asked  to 
respond  to  each  question  using  the  Likert  scale  listed 
below. 

STRONGLY  AGREE  NEUTRAL  DISAGREE  STRONGLY 

AGREE  DISAGREE 


1.  Alternatives  to  delivered  data  are  not  adequately 
or  full  considered,  nor  is  the  data  accession  list 
properly  used.  This  leads  to  higher  costs  and 
difficulties  in  storing  and  retrieving  data. 

2.  The  data  accession  list  is  not  used  to  its  best 
advantage . 

3.  Too  much  unnecessary  information  is  requested  in 
RFP  proposals. 

4.  The  proliferation  of  data  requirements  leads  to 
increased  costs  and  wasted  administrative  effort  for 
both  contractor  and  government. 

5.  Program  requirements  which  are  poorly  defined, 
unstable,  inconsistent,  incomplete,  or  over  specified 
during  the  RFP  preparation  phase  can  lead  to 
inappropriate  data  requirements. 

6.  Government/contractor  efforts  expended  on  data 
pricing  exceed  the  value  added-bene f l ts . 

7.  There  is  a  lack  of  guidance  and  training  in 
determining,  in  determining,  defining,  and  tailoring 
data  requirements. 

8.  There  is  a  lack  of  guidance  and  training  in 
determining,  in  determining,  defining,  and  tailoring  of 
regulations,  specifications,  standards,  etc. 

9.  Because  there  is  no  penalty  for  requesting 
unnecessary  data  (a  lack  of  validated  justification) 
and  no  incentive  to  tailor  data  requirements,  there  is 
a  lack  of  accountability/responsibility  on  the  data 
requester . 

10.  There  are  few  effective  incentives  for  contractors 
to  recommend  data  elimination,  tailoring,  or 
alternatives  to  meet  government  requirements. 
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11.  The  data  acquisition  process  is  inefficient, 
inconsistent,  and/or  poorly  executed  and  contributes  to 
poor  quality  RFPs . 

12.  Poor  communications  exist  between  the  Program 
Manager,  team  members,  supporting  organizations (both 
internal  and  external),  and  contractors,  which 
contributes  to  identifying  data  requirements  in  an 
untimely  manner. 


13.  Often  lessons  learned  with  respect  to  data,  data 
calls,  and  the  tailoring  processes  are  not  documented. 
If  generated  they  are  often  not  used  in  the  preparation 
of  RFPs. 

14.  Existing  policies  and  procedures  do  not 
incentivize  SPOs  and  staffs  to  streamline  data 
requirements . 

15.  Current  policies  and  procedures  do  not  incentivize 
SPOs  and  staffs  to  streamline  data  requirements. 

16.  Current  policies  and  procedures  do  not  incentivize 
the  contractor  (offerors)  to  recommend  streamlined  data 
requirements . 

17.  Lack  of  attention  to  data  management  issues  has 
caused  obsolete  requirements  and  unnecessary  data  to  be 
placed  on  contract. 
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Appendix  C :  Teal  Survey  Suggested  Improvements 


The  Teal  survey  contained  the  30  suggested  improvements 
listed  below  (28).  Those  surveyed  were  asked  to  respond  to 
the  survey  using  the  Likert  scale  listed  immediately  below. 

VERY  HIGH  HIGH  MODERATE  LOW  NONE 

1.  Validate  demonstrated  contractor  capability  through 
CPARS  in  lieu  of  requesting  management  plans  in 
proposals . 

2.  Establish  policy  for  realistic  limits  on  government 
requests  for  alternative/option  pricing  and  cost  data. 

3.  Emphasize  digital  delivery  of  cost  and  pricing 
data,  proposal  data,  and  other  deliverable  data. 

4.  Require  contract  proposals  to  identify  data  that 
would  be  more  economically  provided  in  commercial 
format . 

5.  Provide  an  RFP  preparation  guidance 
document/computerized  system  with  lessons  learned  to 
help  in  preparation  of  RFP. 

6.  Develop  and  define  a  process  at  each  functional 
organization  to  determine  "minimum  essential  data"  to 
do  their  job,  with  responsibility  and  accountability 
placed  on  requester  and  requesting  organization.  In 
practice,  minimizing  data  is  a  responsibility  of  the 
DRRB  rather  than  that  of  the  data  requester  or  his 
organization  even  though  the  latter  officially  does 
have  that  responsibility  also. 

7.  Emphasize  the  many  alternatives  (on-site  visits. 
CAO,  DAL,  etc.)  to  reviewing  contractor 
performance/data  without  necessarily  requesting  data 
for  formal  delivery  to  government. 

8.  Adapt  "mil-prime"  specification  concept  endorsed  by 
MIL-HDBK-248B,  "Acquisition  Streamlining,"  which  allows 
contractor  tailoring  of  government  guidance 
specifications  to  their  design  while  meeting  specified 
operational/performance  requirements . 

9.  Bring  in  outside  organization  (with  identified  team 
players)  into  the  RFP  development  process  as  early  as 
possible . 
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10.  Validate  the  need  for  requirements  (e.g. 
standards,  specs)  derived  from  referenced  documents  and 
require  only  those  that  are  justifiable.  Tailor  to 
program  needs . 

11.  Schedule  has  been  a  major  reason  management  has 
been  willing  to  accept  poorly  defined  data 
requirements.  Management  must  realistically  schedule 
enough  time  for  data  requirements  definition. 

12.  Establish  a  cost  accounting  standard  (CAS) 
guideline  for  pricing  data. 

15.  Develop  an  automated  AMSDL  database  for  direct- 
downloading  to  product  divisions  with  periodic  updates, 
key  word  lookup  capability,  tailoring  guidelines  for 
each  data  item  description  (DID)  and  a  DID  download 
capabi 1 lty . 

14.  Develop  mandatory  training  courses  for  data 
managers  and  RFP  writers. 

15.  Create  career  paths,  recognition  programs  etc., 
for  data  managers. 

16.  Revise  MIL-HDBK-245  to  require  draft  SOWs  to 
clearly  state  mission  performance  criteria;  operational 
effectiveness  measures;  and  operational  suitability 
parameters  in  order  to  avoid  "how-to-manage " 
procedures . 

17.  Ensure  wording  is  included  in  SOW  to  implement  the 
acquisition  streamlining  clause. 

18.  Require  the  use  of  data  accession  list  for  major 
systems . 

19.  Separately  price  each  data  item. 

20.  Increase  approval  level  for  exceptions  to 
separately  pricing  data  for  added  visibility. 

21.  Devise  automated  centralized  systems  for 
recordi ng/ana lyzing  data  prices. 

22.  Establish  a  standard  data  package/draft  CDRL  that 
is  common  to  most  or  all  programs  within  a  product 
division. 

23.  Establish  a  senior-level  data  review  to  act  as  a 
devil's  advocate  review  to  challenge  data  requirements. 
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24.  Implement  "sunset  law"  for  automatic  expiration  in 
AMSDL  and  regulations. 

25.  Prepare  an  AFSC  Data  Management  Guide. 

26.  Recommend  each  product  division  set  up  DM  matrix 
organization . 

27.  Establish  a  data  management  training  program  with 
industry . 

28.  Recommend  AFSC  develop  a  computer-aided  guide  on 
how  to  tailor  data. 

29.  Provide  additional  training  to  those  involved  in 
specifying,  reviewing,  inspecting,  and  accepting 
engineering  data. 

30.  Primary  data  user  should  request  data. 


jendix  D:  Conclusions  of  Teal  Survey 


’inators 


The  following  is  a  summary  of  the  Teal  survey  findings 
the  Tailoring  Data  and  Data  Call  Critical  Action  Team 
presented  to  Head  Quarters  Air  Force  Systems  Command  (1). 

1.  Overall  Problem  Statement:  Tailoring  and  data  call 
processes  need  improvement  in  order  to  ensure  we  meet  our 
user's  requirements. 

2.  All  of  the  problems  identified  in  the  survey  were 
validated  as  problems  via  survey  responses.  Over  80%  of 
those  responding  agreed  that  statements  2.  4.  5.  7,  8,  10, 
13,  and  16  were  problems,  while  the  remaining  statements 
were  identified  as  problems  by  at  least  58%  of  the 
respondees.  The  critical  action  team  (CAT)  therefore 
concluded  all  17  statements  represented  serious  data 
management  problems  (1:13). 


The  most  severe  problems  (top  ten) ,  in  ord 
least  severe,  were  statement  numbers 


of  most  to 


5,  7,  9,  4,  8,  1,  10,  12.  3.  and  14. 

Key  problems  identified  by  the  CAT  were  (1:15): 

a.  Program  requirements  poorly  defined; 

b.  Lack  of  guidance/training  on  data  requirements. 

c.  Lack  of  accountability/responsibility  on  the  data 
requester . 

d.  Proliferation  of  data  requirements. 

e.  Lack  of  guidance/training  on  regulations, 
specifications,  and  standards. 

f.  Not  enough  effort  is  placed  on  pricing  data. 

The  improvements  (top  ten)  most  likely  to  be  beneficial 
to  the  data  acquisition  process  were,  in  order  of  most 
to  least  beneficial,  suggested  improvement  numbers: 


and  15. 


6.  The  CAT  identified  the  following  list  of  improvements 
that  enhance  the  data  acquisition  process  (1:17). 

a.  Order  the  minimum  essential  data  via  one  DRRB. 

b.  Mandatory  training  classes  for  those  involved  in 
data  management.  i.e.  Data  managers,  RFP  writers, 
&  data  reviewers. 

c.  Validate  the  need  for  all  data  requests  and  RFP 
requirements . 

d.  Encourage  the  use  of  commercial/contractor  format 
for  data  delivery. 

e.  Encourage  digital  delivery  of  data. 

f.  Schedule  enough  time  for  requirements  definition. 

g.  Provide  additional  training. 
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Appendix  E :  Kolmoqorov-Smirnov  Test  on  Tea  1  Data 

The  following  contains  the  results  of  the  Kolmogorov- 
Smirnov  (K-S)  test  on  the  Teal  survey  data.  The  responses 
of  AFLC  and  ASD  to  each  question  were  tested  for 
similarities  of  spread  and  shape  of  their  distributions. 
Determining  that  the  spread  and  shape  of  the  distributions 
were  the  close  to  the  same  enabled  the  Wilcoxon  Rank-Sum 
Test  to  be  used  for  subsequent  tests.  The  P-values  of  the 
test  are  reported.  Any  P-values  less  than  0.20  will  show 
different  distributions. 


PROBLEM 

IMPROVEMENT 

STATEMENT 

P-va 1 ue 

STATEMENT 

P-va 1 ue 

1 

.8849 

1 

1 . 0000 

2 

1.0000 

2 

.5537 

3 

1.0000 

3 

1 . 0000 

4 

1.0000 

4 

1 . 0000 

5 

1.0000 

5 

1 . 0000 

6 

1 . 0000 

6 

.6226 

7 

1.0000 

7 

.6534 

8 

1.0000 

8 

1.0000 

9 

.8638 

9 

1.0000 

10 

1.0000 

10 

1.0000 

11 

1.0000 

11 

1.0000 

12 

1 . 0000 

12 

1.0000 

13 

1.0000 

13 

1.0000 

14 

1 . 0000 

14 

1 . 0000 

15 

1 . 0000 

15 

1 .0000 

16 

1.0000 

16 

1 . 0000 

17 

.9110 

17 

1 . 0000 

18 

1 . 0000 

19 

1 . 0000 

20 

1 . 0000 

21 

.  2301 

22 

1 .0000 

23 

.  1599 

24 

1.0000 

25 

1 . 0000 

26 

1 . 0000 

27 

1.0000 

28 

1.0000 

29 

1 . 0000 

30 

1.0000 

(P-values 

are  for  a 

two-tailed  test.) 
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APPENDIX  F:  Factor  Analysis  of  Teal  Survey  Problems  and  Improvements 

This  is  the  factor  analysis  results  when  the  top  ranked  responses  of 
problems  and  suggested  improvements  from  the  Teal  Survey  are  analysed  as 
a  group.  Listed  below  is  the  SPSS-X  program  used  and  the  resulting 
rotated  factor  matrix  is  shown  on  the  following  page.  Only  the  highest 
loading  factor  for  each  variable  is  shown  in  the  factor  analysis  matrix. 


TITLE 

FILE  HANDLE 
DATA  LIST 


SET 

MISSING  VALUES 
FACTOR 


'SPSSX  ON  TEALE  DATA' 

DAT1/NAME- ' TEALEDAT ' 

FILE-DAT1  FIXED  RECORDS-1/ 

ID . PR1 , PR2 . PR3 . PR4 . PR5 , PR6 , PR7 , PR8 , PR9 . 

PR10 . PR11 . PR12 . PR13 . PR14 . PR15 . PR16 . PR17 , IM1 , IM2 . IM3 . 
IM4 ,  IM5 .  IM6 .  IM7 .  IM8 ,  IM9 ,  IM10 ,  IM11 ,  IM12 .  IM13 ,  IM14 ,  IM15 , 
IM16 , IM17 . IM18 . IM1 9 . IM20 . IM21 , IM22 , IM23 , IM24 , IM25 , 

IM26 , IM27 , IM28 . IM29 . IM30 
(F2.0.47F1.0) 

ELANK=99/WIDTH=80 
ID  to  IM30  (991 

VARIABLES=PR4 ,PR5,PR7, PR8 . PR1 0 . PR1 3 , PR16 . PR17 , IM4 , IM5 , 
IM10 . IM11 . IM14. IM29 . IM30/ 


FORMAT-SORT  BLANK  (.3)/ 


FINAL  STATISTICS: 


VARIABLE 

COMMUNALITY 

* 

* 

FACTOR 

EIGENVALUE 

PCT  OF  VAR 

CUM  PCT 

PR4 

. 66908 

* 

1 

4.14061 

27.6 

27.6 

PR5 

.61711 

★ 

2 

2 . 50952 

16.7 

44.3 

PR7 

.78610 

* 

3 

1.72046 

11.5 

55.8 

PR8 

.71125 

★ 

4 

1.13602 

7.6 

63.4 

PR10 

.54890 

★ 

5 

1 . 08858 

7.3 

70.6 

PR13 

.77346 

★ 

PR16 

.63201 

* 

PR17 

.76447 

* 

IM4 

.52583 

★ 

IMS 

.85727 

★ 

IM10 

.71644 

* 

IM11 

.84152 

★ 

IM14 

.79664 

★ 

IM29 

.72553 

★ 

IM30 

.62960 

★ 
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VARIMAX 

ROTATION  1  FOR  EXTRACTION  1  IN  ANALYSIS  1  -  KAISER  NORMALIZATION. 
VARIMAX  CONVERGED  IN  8  ITERATIONS. 


ROTATED  FACTOR  MATRIX: 


FACTOR  1 

FACTOR 

PR7 

.83943 

PR8 

.81637 

IM14 

.73128 

PR4 

.56221 

IM29 

.56042 

IM5 

.76780 

IM30 

.76132 

IM10 

.71159 

IM4 

.68839 

PR16 

PR10 

PR17 

IM11 

PR5 

PR13 

FACTOR  3  FACTOR  4  FACTOR  5 


.74204 

.72193 

.62367 

.90394 

.61773 

.83113 


FACTOR  TRANSFORMATION  MATRIX: 


FACTOR  1 
FACTOR  2 
FACTOR  3 
FACTOR  4 
FACTOR  5 


FACTOR  1 

.66173 
-.47491 
-.50704 
- . 15839 
-.23323 


FACTOR  2 

.45278 

.72883 

-.01764 

.33522 

-.38873 


FACTOR  3 

.40172 
-.24519 
. 86073 
-.13062 
-.14349 


FACTOR  4 

. 37296 
.36473 
-.02659 
-.48406 
.70203 


FACTOR  5 

. 23797 
- . 22387 
.03218 
.78177 
.53015 


For  more  information  on  factor  analysis  and  rotated  factor  matrix, 
consult  Hnory's  Business  Research  Methods  (13:402-407),  and 
Kachigan's  Statistical  Analysis  (19:377-401). 
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